2024-08-30T15:47:03.661 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-08-30T15:47:03.679 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-30T15:47:03.778 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi/7881803 branch: wip-ksirivad-backport-reef-57906 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/valgrind} email: null first_in_suite: false job_id: '7881803' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-ksirivad-backport-reef-57906 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 osd op thread timeout: 150 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(OSDMAP_FLAGS\) - \(SMALLER_PGP_NUM\) - \(POOL_APP_NOT_ENABLED\) - \(SLOW OPS\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: e5753ca846ae583f991bb12d29862578d4f67590 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: debuginfo: true flavor: default sha1: e5753ca846ae583f991bb12d29862578d4f67590 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-ksirivad-backport-reef-57906 sha1: e5753ca846ae583f991bb12d29862578d4f67590 owner: scheduled_ksirivad@teuthology priority: 70 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 819 sha1: e5753ca846ae583f991bb12d29862578d4f67590 sleep_before_teardown: 0 subset: 90000/120000 suite: rados suite_branch: wip-ksirivad-backport-reef-57906 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_e5753ca846ae583f991bb12d29862578d4f67590/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: e5753ca846ae583f991bb12d29862578d4f67590 targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFI5hI4j98WMgLeNrcLdzDCoQr7AzKV7z6rlHx2+q6qRSRxBzKBR03RGmTdppxhI4ZuBK6bjlA2e+GFjedqrb1s= smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJM6mdZQusRlJDgXQK2y23aASpUg/lnKj8AM+uOaNq0kEIkRNsTgsUnnxnbVULa7yTGKzo/F3pUBpaGZo8TfjDA= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - mon_recovery: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-08-30_14:53:30 tube: smithi user: ksirivad verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-30T15:47:03.778 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_e5753ca846ae583f991bb12d29862578d4f67590/qa; will attempt to use it 2024-08-30T15:47:03.779 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_e5753ca846ae583f991bb12d29862578d4f67590/qa/tasks 2024-08-30T15:47:03.779 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-30T15:47:03.781 INFO:teuthology.task.internal:Checking packages... 2024-08-30T15:47:03.801 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e5753ca846ae583f991bb12d29862578d4f67590' 2024-08-30T15:47:03.801 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-30T15:47:03.801 INFO:teuthology.packaging:ref: None 2024-08-30T15:47:03.802 INFO:teuthology.packaging:tag: None 2024-08-30T15:47:03.802 INFO:teuthology.packaging:branch: wip-ksirivad-backport-reef-57906 2024-08-30T15:47:03.802 INFO:teuthology.packaging:sha1: e5753ca846ae583f991bb12d29862578d4f67590 2024-08-30T15:47:03.802 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-ksirivad-backport-reef-57906 2024-08-30T15:47:03.957 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-799.ge5753ca8 2024-08-30T15:47:03.959 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-30T15:47:03.965 INFO:teuthology.task.internal:no buildpackages task found 2024-08-30T15:47:03.965 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-30T15:47:03.974 INFO:teuthology.task.internal:Saving configuration 2024-08-30T15:47:03.988 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-30T15:47:03.995 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-30T15:47:04.030 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi/7881803', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-30 15:42:12.654358', 'locked_by': 'scheduled_ksirivad@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFI5hI4j98WMgLeNrcLdzDCoQr7AzKV7z6rlHx2+q6qRSRxBzKBR03RGmTdppxhI4ZuBK6bjlA2e+GFjedqrb1s='} 2024-08-30T15:47:04.063 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi/7881803', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-30 15:42:12.652958', 'locked_by': 'scheduled_ksirivad@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJM6mdZQusRlJDgXQK2y23aASpUg/lnKj8AM+uOaNq0kEIkRNsTgsUnnxnbVULa7yTGKzo/F3pUBpaGZo8TfjDA='} 2024-08-30T15:47:04.063 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-30T15:47:04.069 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-08-30T15:47:04.070 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-08-30T15:47:04.070 INFO:teuthology.run_tasks:Running task console_log... 2024-08-30T15:47:04.147 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9a9de97be0>, signals=[15]) 2024-08-30T15:47:04.147 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-30T15:47:04.153 INFO:teuthology.task.internal:Opening connections... 2024-08-30T15:47:04.153 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2024-08-30T15:47:04.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T15:47:04.239 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2024-08-30T15:47:04.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T15:47:04.323 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-30T15:47:04.329 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2024-08-30T15:47:04.347 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2024-08-30T15:47:04.347 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2024-08-30T15:47:04.405 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2024-08-30T15:47:04.405 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="9" 2024-08-30T15:47:04.405 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2024-08-30T15:47:04.405 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2024-08-30T15:47:04.405 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="9" 2024-08-30T15:47:04.405 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el9" 2024-08-30T15:47:04.405 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-30T15:47:04.406 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2024-08-30T15:47:04.406 INFO:teuthology.orchestra.run.smithi050.stdout:LOGO="fedora-logo-icon" 2024-08-30T15:47:04.406 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-30T15:47:04.406 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2024-08-30T15:47:04.406 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-30T15:47:04.406 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-30T15:47:04.406 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-30T15:47:04.407 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2024-08-30T15:47:04.436 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2024-08-30T15:47:04.454 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2024-08-30T15:47:04.455 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2024-08-30T15:47:04.513 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Stream" 2024-08-30T15:47:04.513 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="9" 2024-08-30T15:47:04.514 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2024-08-30T15:47:04.514 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2024-08-30T15:47:04.514 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="9" 2024-08-30T15:47:04.514 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el9" 2024-08-30T15:47:04.514 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-30T15:47:04.514 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2024-08-30T15:47:04.514 INFO:teuthology.orchestra.run.smithi094.stdout:LOGO="fedora-logo-icon" 2024-08-30T15:47:04.514 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-30T15:47:04.514 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2024-08-30T15:47:04.514 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-30T15:47:04.514 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-30T15:47:04.514 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-30T15:47:04.515 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2024-08-30T15:47:04.544 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-30T15:47:04.553 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-30T15:47:04.559 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-30T15:47:04.559 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2024-08-30T15:47:04.562 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2024-08-30T15:47:04.579 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-30T15:47:04.585 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-30T15:47:04.585 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2024-08-30T15:47:04.619 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2024-08-30T15:47:04.739 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-30T15:47:04.813 INFO:teuthology.run_tasks:Running task kernel... 2024-08-30T15:47:04.829 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-30T15:47:04.829 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-30T15:47:04.829 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-30T15:47:04.830 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-30T15:47:04.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T15:47:04.847 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-08-30T15:47:04.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T15:47:04.848 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-08-30T15:47:04.905 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-503.el9.x86_64 2024-08-30T15:47:04.906 INFO:teuthology.task.kernel:Running kernel on smithi094: 5.14.0-503.el9.x86_64 2024-08-30T15:47:04.906 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2024-08-30T15:47:04.907 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-503.el9.x86_64 2024-08-30T15:47:04.907 INFO:teuthology.task.kernel:Running kernel on smithi050: 5.14.0-503.el9.x86_64 2024-08-30T15:47:04.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2024-08-30T15:47:07.550 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 8.8 MB/s | 8.2 MB 00:00 2024-08-30T15:47:09.814 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.2 MB 00:03 2024-08-30T15:47:09.934 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 17 MB/s | 20 MB 00:01 2024-08-30T15:47:11.565 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-08-30T15:47:16.240 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.6 MB 00:00 2024-08-30T15:47:16.451 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.6 MB 00:02 2024-08-30T15:47:17.841 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 18 kB 00:00 2024-08-30T15:47:17.869 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 109 kB/s | 18 kB 00:00 2024-08-30T15:47:19.478 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 15 MB/s | 22 MB 00:01 2024-08-30T15:47:19.531 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 15 MB/s | 22 MB 00:01 2024-08-30T15:47:24.834 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-08-30T15:47:25.137 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 49 kB/s | 1.7 kB 00:00 2024-08-30T15:47:26.857 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-30T15:47:26.857 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-08-30T15:47:26.935 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:47:26.936 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-08-30T15:47:26.937 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:47:27.033 DEBUG:teuthology.orchestra.run.smithi094:> echo no | sudo yum reinstall kernel || true 2024-08-30T15:47:27.277 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-30T15:47:27.277 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-08-30T15:47:27.360 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:47:27.361 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-08-30T15:47:27.361 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:47:27.484 DEBUG:teuthology.orchestra.run.smithi050:> echo no | sudo yum reinstall kernel || true 2024-08-30T15:47:27.572 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:03 ago on Fri 30 Aug 2024 03:47:24 PM UTC. 2024-08-30T15:47:27.705 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:47:27.711 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:47:27.712 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-08-30T15:47:27.712 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:47:27.712 INFO:teuthology.orchestra.run.smithi094.stdout:Reinstalling: 2024-08-30T15:47:27.712 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-08-30T15:47:27.712 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:47:27.712 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:47:27.712 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:47:27.712 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:47:27.712 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 1.9 M 2024-08-30T15:47:27.712 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 0 2024-08-30T15:47:27.713 INFO:teuthology.orchestra.run.smithi094.stderr:Operation aborted. 2024-08-30T15:47:27.769 INFO:teuthology.orchestra.run.smithi094.stdout:Is this ok [y/N]: 2024-08-30T15:47:27.769 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum reinstall -y kernel || true 2024-08-30T15:47:28.079 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:03 ago on Fri 30 Aug 2024 03:47:25 PM UTC. 2024-08-30T15:47:28.227 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:47:28.237 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:47:28.237 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-08-30T15:47:28.237 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:47:28.237 INFO:teuthology.orchestra.run.smithi050.stdout:Reinstalling: 2024-08-30T15:47:28.238 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-08-30T15:47:28.238 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:47:28.238 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:47:28.238 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:47:28.238 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:47:28.238 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 1.9 M 2024-08-30T15:47:28.238 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 0 2024-08-30T15:47:28.238 INFO:teuthology.orchestra.run.smithi050.stderr:Operation aborted. 2024-08-30T15:47:28.295 INFO:teuthology.orchestra.run.smithi050.stdout:Is this ok [y/N]: 2024-08-30T15:47:28.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum reinstall -y kernel || true 2024-08-30T15:47:28.300 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:04 ago on Fri 30 Aug 2024 03:47:24 PM UTC. 2024-08-30T15:47:28.425 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:47:28.426 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:47:28.426 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-08-30T15:47:28.426 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:47:28.426 INFO:teuthology.orchestra.run.smithi094.stdout:Reinstalling: 2024-08-30T15:47:28.426 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-08-30T15:47:28.426 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:47:28.427 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:47:28.427 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:47:28.427 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:47:28.427 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 1.9 M 2024-08-30T15:47:28.427 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 0 2024-08-30T15:47:28.427 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:47:28.843 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:03 ago on Fri 30 Aug 2024 03:47:25 PM UTC. 2024-08-30T15:47:28.974 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:47:28.975 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:47:28.975 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-08-30T15:47:28.975 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:47:28.975 INFO:teuthology.orchestra.run.smithi050.stdout:Reinstalling: 2024-08-30T15:47:28.975 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-08-30T15:47:28.975 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:47:28.975 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:47:28.975 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:47:28.975 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:47:28.975 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 1.9 M 2024-08-30T15:47:28.975 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 0 2024-08-30T15:47:28.975 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:47:29.103 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-503.el9.x86_64.rpm 4.4 MB/s | 1.9 MB 00:00 2024-08-30T15:47:29.103 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:47:29.103 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.8 MB/s | 1.9 MB 00:00 2024-08-30T15:47:29.135 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:47:29.281 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-503.el9.x86_64.rpm 9.0 MB/s | 1.9 MB 00:00 2024-08-30T15:47:29.281 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:47:29.281 INFO:teuthology.orchestra.run.smithi050.stdout:Total 6.1 MB/s | 1.9 MB 00:00 2024-08-30T15:47:29.320 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:47:29.374 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:47:29.374 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:47:29.405 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:47:29.405 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:47:29.578 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:47:29.578 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:47:29.586 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:47:29.610 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:47:29.610 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:47:29.672 INFO:teuthology.orchestra.run.smithi094.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-08-30T15:47:29.672 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T15:47:29.791 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:47:29.883 INFO:teuthology.orchestra.run.smithi050.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-08-30T15:47:29.884 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T15:47:30.461 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T15:47:30.461 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-08-30T15:47:30.585 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T15:47:30.585 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-08-30T15:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T15:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:47:31.207 INFO:teuthology.orchestra.run.smithi094.stdout:Reinstalled: 2024-08-30T15:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-5.14.0-503.el9.x86_64 2024-08-30T15:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:47:31.208 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:47:31.347 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T15:47:31.347 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:47:31.347 INFO:teuthology.orchestra.run.smithi050.stdout:Reinstalled: 2024-08-30T15:47:31.347 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-5.14.0-503.el9.x86_64 2024-08-30T15:47:31.347 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:47:31.347 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:47:31.391 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-08-30T15:47:31.450 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-503.el9.x86_64 2024-08-30T15:47:31.450 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-08-30T15:47:31.451 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-30T15:47:31.451 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-08-30T15:47:31.451 DEBUG:teuthology.orchestra.run.smithi094:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-30T15:47:31.488 INFO:teuthology.orchestra.run.smithi094.stdout:ttyS1 2024-08-30T15:47:31.507 DEBUG:teuthology.parallel:result is None 2024-08-30T15:47:31.527 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-08-30T15:47:31.585 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-503.el9.x86_64 2024-08-30T15:47:31.586 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-08-30T15:47:31.586 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-30T15:47:31.586 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-30T15:47:31.586 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-30T15:47:31.619 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2024-08-30T15:47:31.639 DEBUG:teuthology.parallel:result is None 2024-08-30T15:47:31.639 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-30T15:47:31.646 INFO:teuthology.task.internal:Creating test directory... 2024-08-30T15:47:31.646 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-30T15:47:31.682 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-30T15:47:31.701 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-30T15:47:31.707 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-30T15:47:31.716 INFO:teuthology.task.internal:Creating archive directory... 2024-08-30T15:47:31.716 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-30T15:47:31.737 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-30T15:47:31.768 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-30T15:47:31.775 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-30T15:47:31.775 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-30T15:47:31.803 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-30T15:47:31.829 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T15:47:31.839 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T15:47:31.853 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T15:47:31.864 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T15:47:31.865 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-30T15:47:31.872 INFO:teuthology.task.internal:Configuring sudo... 2024-08-30T15:47:31.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-30T15:47:31.883 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-30T15:47:31.908 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-30T15:47:31.917 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-30T15:47:31.917 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-30T15:47:31.948 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-30T15:47:31.963 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-30T15:47:32.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-30T15:47:32.101 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-30T15:47:32.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-30T15:47:32.223 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:47:32.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-30T15:47:32.287 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-30T15:47:32.334 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-30T15:47:32.379 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-30T15:47:32.466 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-30T15:47:32.501 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:47:32.501 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-30T15:47:32.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2024-08-30T15:47:32.568 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2024-08-30T15:47:32.604 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-30T15:47:32.641 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-30T15:47:33.159 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-30T15:47:33.167 INFO:teuthology.task.internal:Starting timer... 2024-08-30T15:47:33.167 INFO:teuthology.run_tasks:Running task pcp... 2024-08-30T15:47:33.179 INFO:teuthology.run_tasks:Running task selinux... 2024-08-30T15:47:33.188 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-08-30T15:47:33.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2024-08-30T15:47:33.229 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2024-08-30T15:47:33.231 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2024-08-30T15:47:33.266 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: 2024-08-30T15:47:33.268 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-30T15:47:33.268 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2024-08-30T15:47:33.293 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2024-08-30T15:47:33.293 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2024-08-30T15:47:33.334 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2024-08-30T15:47:33.334 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi050.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi094.front.sepia.ceph.com': 'permissive'} 2024-08-30T15:47:33.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-30T15:47:33.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T15:47:33.359 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-30T15:47:33.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T15:47:33.400 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-30T15:47:33.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2024-08-30T15:47:33.433 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2024-08-30T15:47:33.472 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-30T15:47:33.494 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-30T15:47:33.505 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-30T15:47:33.507 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com,smithi094.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-30T15:49:17.666 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi050.front.sepia.ceph.com'), Remote(name='ubuntu@smithi094.front.sepia.ceph.com')] 2024-08-30T15:49:17.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-08-30T15:49:17.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T15:49:17.753 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-08-30T15:49:17.832 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-08-30T15:49:17.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-08-30T15:49:17.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T15:49:17.917 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-08-30T15:49:18.000 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-08-30T15:49:18.000 INFO:teuthology.run_tasks:Running task clock... 2024-08-30T15:49:18.011 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-30T15:49:18.011 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-30T15:49:18.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-30T15:49:18.014 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-30T15:49:18.014 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-30T15:49:18.042 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-30T15:49:18.057 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-30T15:49:18.068 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-30T15:49:18.081 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2024-08-30T15:49:18.083 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-30T15:49:18.092 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2024-08-30T15:49:18.107 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2024-08-30T15:49:18.108 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-30T15:49:18.118 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2024-08-30T15:49:18.120 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-30T15:49:18.132 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-30T15:49:18.146 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-30T15:49:18.181 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-08-30T15:49:18.184 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-30T15:49:18.184 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-08-30T15:49:18.184 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T15:49:18.184 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T15:49:18.185 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T15:49:18.185 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T15:49:18.205 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-08-30T15:49:18.209 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-30T15:49:18.209 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-08-30T15:49:18.209 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T15:49:18.209 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T15:49:18.209 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T15:49:18.209 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T15:49:18.210 INFO:teuthology.run_tasks:Running task install... 2024-08-30T15:49:18.218 DEBUG:teuthology.task.install:project ceph 2024-08-30T15:49:18.218 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'e5753ca846ae583f991bb12d29862578d4f67590'}} 2024-08-30T15:49:18.218 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'e5753ca846ae583f991bb12d29862578d4f67590'} 2024-08-30T15:49:18.218 INFO:teuthology.task.install:Using flavor: default 2024-08-30T15:49:18.223 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-08-30T15:49:18.224 INFO:teuthology.task.install:extra packages: [] 2024-08-30T15:49:18.224 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e5753ca846ae583f991bb12d29862578d4f67590', 'tag': None, 'wait_for_package': False} 2024-08-30T15:49:18.224 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e5753ca846ae583f991bb12d29862578d4f67590 2024-08-30T15:49:18.226 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e5753ca846ae583f991bb12d29862578d4f67590', 'tag': None, 'wait_for_package': False} 2024-08-30T15:49:18.227 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e5753ca846ae583f991bb12d29862578d4f67590 2024-08-30T15:49:18.378 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-ksirivad-backport-reef-57906/e5753ca846ae583f991bb12d29862578d4f67590/centos/9/flavors/default/ 2024-08-30T15:49:18.378 INFO:teuthology.task.install.rpm:Package version is 18.2.4-799.ge5753ca8 2024-08-30T15:49:18.390 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-ksirivad-backport-reef-57906/e5753ca846ae583f991bb12d29862578d4f67590/centos/9/flavors/default/ 2024-08-30T15:49:18.390 INFO:teuthology.task.install.rpm:Package version is 18.2.4-799.ge5753ca8 2024-08-30T15:49:18.494 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-ksirivad-backport-reef-57906/e5753ca846ae583f991bb12d29862578d4f67590/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-ksirivad-backport-reef-57906/e5753ca846ae583f991bb12d29862578d4f67590/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-ksirivad-backport-reef-57906/e5753ca846ae583f991bb12d29862578d4f67590/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-30T15:49:18.494 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:49:18.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-30T15:49:18.516 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-ksirivad-backport-reef-57906/e5753ca846ae583f991bb12d29862578d4f67590/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-ksirivad-backport-reef-57906/e5753ca846ae583f991bb12d29862578d4f67590/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-ksirivad-backport-reef-57906/e5753ca846ae583f991bb12d29862578d4f67590/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-30T15:49:18.516 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:49:18.516 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-30T15:49:18.521 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-08-30T15:49:18.521 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e5753ca846ae583f991bb12d29862578d4f67590/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-30T15:49:18.542 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-08-30T15:49:18.542 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e5753ca846ae583f991bb12d29862578d4f67590/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-30T15:49:18.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-30T15:49:18.608 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-30T15:49:18.661 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-30T15:49:18.683 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-30T15:49:18.724 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2024-08-30T15:49:18.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2024-08-30T15:49:18.748 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2024-08-30T15:49:18.749 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2024-08-30T15:49:18.956 INFO:teuthology.orchestra.run.smithi050.stdout:45 files removed 2024-08-30T15:49:18.972 INFO:teuthology.orchestra.run.smithi094.stdout:45 files removed 2024-08-30T15:49:18.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw 2024-08-30T15:49:18.998 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw 2024-08-30T15:49:19.536 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 292 kB/s | 77 kB 00:00 2024-08-30T15:49:19.540 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 282 kB/s | 77 kB 00:00 2024-08-30T15:49:19.816 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 50 kB/s | 11 kB 00:00 2024-08-30T15:49:19.887 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 37 kB/s | 11 kB 00:00 2024-08-30T15:49:20.083 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-08-30T15:49:20.154 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-08-30T15:49:20.596 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.2 MB 00:00 2024-08-30T15:49:22.158 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.2 MB 00:02 2024-08-30T15:49:23.747 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 20 MB 00:02 2024-08-30T15:49:25.609 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 9.0 MB/s | 20 MB 00:02 2024-08-30T15:49:29.322 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.6 MB 00:01 2024-08-30T15:49:30.965 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 18 kB 00:00 2024-08-30T15:49:30.985 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.6 MB 00:01 2024-08-30T15:49:32.127 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:01 2024-08-30T15:49:33.469 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 18 kB 00:01 2024-08-30T15:49:35.041 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 16 MB/s | 22 MB 00:01 2024-08-30T15:49:37.466 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-30T15:49:39.791 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:49:39.794 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:49:39.794 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-08-30T15:49:39.794 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:49:39.794 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:49:39.794 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 7.7 M 2024-08-30T15:49:39.794 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2024-08-30T15:49:39.794 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 3.3 M 2024-08-30T15:49:39.794 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 3.0 M 2024-08-30T15:49:39.794 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-08-30T15:49:39.794 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-30T15:49:39.794 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 5.1 M 2024-08-30T15:49:39.795 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 18 M 2024-08-30T15:49:39.795 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 25 k 2024-08-30T15:49:39.795 INFO:teuthology.orchestra.run.smithi094.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-08-30T15:49:39.795 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-30T15:49:39.795 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-30T15:49:39.795 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-30T15:49:39.795 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 711 k 2024-08-30T15:49:39.795 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-30T15:49:39.795 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-30T15:49:39.795 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-30T15:49:39.795 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 478 k 2024-08-30T15:49:39.796 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-30T15:49:39.796 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 4.5 M 2024-08-30T15:49:39.796 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-30T15:49:39.796 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-30T15:49:39.796 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-30T15:49:39.796 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 45 k 2024-08-30T15:49:39.796 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 129 k 2024-08-30T15:49:39.796 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 162 k 2024-08-30T15:49:39.796 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-30T15:49:39.796 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 320 k 2024-08-30T15:49:39.796 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 298 k 2024-08-30T15:49:39.797 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 99 k 2024-08-30T15:49:39.797 INFO:teuthology.orchestra.run.smithi094.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-08-30T15:49:39.797 INFO:teuthology.orchestra.run.smithi094.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-30T15:49:39.797 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:49:39.797 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:49:39.797 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:49:39.797 INFO:teuthology.orchestra.run.smithi094.stdout:Install 28 Packages 2024-08-30T15:49:39.797 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 2 Packages 2024-08-30T15:49:39.797 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:49:39.798 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 53 M 2024-08-30T15:49:39.798 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:49:40.671 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-08-30T15:49:41.151 INFO:teuthology.orchestra.run.smithi094.stdout:(1/30): ceph-base-18.2.4-799.ge5753ca8.el9.x86_ 6.7 MB/s | 5.1 MB 00:00 2024-08-30T15:49:41.251 INFO:teuthology.orchestra.run.smithi094.stdout:(2/30): ceph-radosgw-18.2.4-799.ge5753ca8.el9.x 9.0 MB/s | 7.7 MB 00:00 2024-08-30T15:49:41.276 INFO:teuthology.orchestra.run.smithi094.stdout:(3/30): ceph-selinux-18.2.4-799.ge5753ca8.el9.x 201 kB/s | 25 kB 00:00 2024-08-30T15:49:41.319 INFO:teuthology.orchestra.run.smithi094.stdout:(4/30): libcephfs2-18.2.4-799.ge5753ca8.el9.x86 10 MB/s | 711 kB 00:00 2024-08-30T15:49:41.344 INFO:teuthology.orchestra.run.smithi094.stdout:(5/30): libradosstriper1-18.2.4-799.ge5753ca8.e 7.0 MB/s | 478 kB 00:00 2024-08-30T15:49:41.552 INFO:teuthology.orchestra.run.smithi094.stdout:(6/30): ceph-common-18.2.4-799.ge5753ca8.el9.x8 16 MB/s | 18 MB 00:01 2024-08-30T15:49:41.577 INFO:teuthology.orchestra.run.smithi094.stdout:(7/30): python3-ceph-argparse-18.2.4-799.ge5753 193 kB/s | 45 kB 00:00 2024-08-30T15:49:41.603 INFO:teuthology.orchestra.run.smithi094.stdout:(8/30): python3-ceph-common-18.2.4-799.ge5753ca 2.5 MB/s | 129 kB 00:00 2024-08-30T15:49:41.628 INFO:teuthology.orchestra.run.smithi094.stdout:(9/30): python3-cephfs-18.2.4-799.ge5753ca8.el9 3.2 MB/s | 162 kB 00:00 2024-08-30T15:49:41.653 INFO:teuthology.orchestra.run.smithi094.stdout:(10/30): python3-rados-18.2.4-799.ge5753ca8.el9 6.2 MB/s | 320 kB 00:00 2024-08-30T15:49:41.678 INFO:teuthology.orchestra.run.smithi094.stdout:(11/30): python3-rbd-18.2.4-799.ge5753ca8.el9.x 5.8 MB/s | 298 kB 00:00 2024-08-30T15:49:41.704 INFO:teuthology.orchestra.run.smithi094.stdout:(12/30): python3-rgw-18.2.4-799.ge5753ca8.el9.x 1.9 MB/s | 99 kB 00:00 2024-08-30T15:49:41.787 INFO:teuthology.orchestra.run.smithi094.stdout:(13/30): librgw2-18.2.4-799.ge5753ca8.el9.x86_6 9.5 MB/s | 4.5 MB 00:00 2024-08-30T15:49:41.838 INFO:teuthology.orchestra.run.smithi094.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 673 kB/s | 107 kB 00:00 2024-08-30T15:49:41.922 INFO:teuthology.orchestra.run.smithi094.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 338 kB/s | 45 kB 00:00 2024-08-30T15:49:41.938 INFO:teuthology.orchestra.run.smithi094.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 683 kB/s | 160 kB 00:00 2024-08-30T15:49:41.989 INFO:teuthology.orchestra.run.smithi094.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 847 kB/s | 42 kB 00:00 2024-08-30T15:49:42.030 INFO:teuthology.orchestra.run.smithi094.stdout:(18/30): fmt-8.1.1-5.el9.x86_64.rpm 2.6 MB/s | 111 kB 00:00 2024-08-30T15:49:42.064 INFO:teuthology.orchestra.run.smithi094.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.9 MB/s | 662 kB 00:00 2024-08-30T15:49:42.089 INFO:teuthology.orchestra.run.smithi094.stdout:(20/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 292 kB 00:00 2024-08-30T15:49:42.131 INFO:teuthology.orchestra.run.smithi094.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.0 MB/s | 309 kB 00:00 2024-08-30T15:49:42.156 INFO:teuthology.orchestra.run.smithi094.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-08-30T15:49:42.182 INFO:teuthology.orchestra.run.smithi094.stdout:(23/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 307 kB/s | 26 kB 00:00 2024-08-30T15:49:42.207 INFO:teuthology.orchestra.run.smithi094.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-08-30T15:49:42.266 INFO:teuthology.orchestra.run.smithi094.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-08-30T15:49:42.383 INFO:teuthology.orchestra.run.smithi094.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.1 MB/s | 839 kB 00:00 2024-08-30T15:49:42.424 INFO:teuthology.orchestra.run.smithi094.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-08-30T15:49:42.516 INFO:teuthology.orchestra.run.smithi094.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 4.4 MB 00:00 2024-08-30T15:49:42.633 INFO:teuthology.orchestra.run.smithi094.stdout:(29/30): librbd1-18.2.4-799.ge5753ca8.el9.x86_6 14 MB/s | 3.0 MB 00:00 2024-08-30T15:49:42.708 INFO:teuthology.orchestra.run.smithi094.stdout:(30/30): librados2-18.2.4-799.ge5753ca8.el9.x86 10 MB/s | 3.3 MB 00:00 2024-08-30T15:49:42.714 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:49:42.714 INFO:teuthology.orchestra.run.smithi094.stdout:Total 18 MB/s | 53 MB 00:02 2024-08-30T15:49:42.786 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:49:42.822 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:49:42.822 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:49:43.090 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:49:43.092 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:49:43.093 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-08-30T15:49:43.093 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:49:43.093 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:49:43.093 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 7.7 M 2024-08-30T15:49:43.093 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2024-08-30T15:49:43.093 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 3.3 M 2024-08-30T15:49:43.093 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 3.0 M 2024-08-30T15:49:43.093 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-08-30T15:49:43.093 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-30T15:49:43.093 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 5.1 M 2024-08-30T15:49:43.093 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 18 M 2024-08-30T15:49:43.093 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 25 k 2024-08-30T15:49:43.094 INFO:teuthology.orchestra.run.smithi050.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-08-30T15:49:43.094 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-30T15:49:43.094 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-30T15:49:43.094 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-30T15:49:43.094 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 711 k 2024-08-30T15:49:43.094 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-30T15:49:43.094 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-30T15:49:43.094 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-30T15:49:43.094 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 478 k 2024-08-30T15:49:43.094 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-30T15:49:43.094 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 4.5 M 2024-08-30T15:49:43.094 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-30T15:49:43.094 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-30T15:49:43.095 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-30T15:49:43.095 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 45 k 2024-08-30T15:49:43.095 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 129 k 2024-08-30T15:49:43.095 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 162 k 2024-08-30T15:49:43.095 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-30T15:49:43.095 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 320 k 2024-08-30T15:49:43.095 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 298 k 2024-08-30T15:49:43.095 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 99 k 2024-08-30T15:49:43.095 INFO:teuthology.orchestra.run.smithi050.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-08-30T15:49:43.095 INFO:teuthology.orchestra.run.smithi050.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-30T15:49:43.095 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:49:43.095 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:49:43.095 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:49:43.096 INFO:teuthology.orchestra.run.smithi050.stdout:Install 28 Packages 2024-08-30T15:49:43.096 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 2 Packages 2024-08-30T15:49:43.096 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:49:43.096 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 53 M 2024-08-30T15:49:43.096 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:49:43.247 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:49:43.247 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:49:43.939 INFO:teuthology.orchestra.run.smithi050.stdout:(1/30): ceph-base-18.2.4-799.ge5753ca8.el9.x86_ 9.1 MB/s | 5.1 MB 00:00 2024-08-30T15:49:44.040 INFO:teuthology.orchestra.run.smithi050.stdout:(2/30): ceph-radosgw-18.2.4-799.ge5753ca8.el9.x 12 MB/s | 7.7 MB 00:00 2024-08-30T15:49:44.065 INFO:teuthology.orchestra.run.smithi050.stdout:(3/30): ceph-selinux-18.2.4-799.ge5753ca8.el9.x 201 kB/s | 25 kB 00:00 2024-08-30T15:49:44.098 INFO:teuthology.orchestra.run.smithi050.stdout:(4/30): libcephfs2-18.2.4-799.ge5753ca8.el9.x86 12 MB/s | 711 kB 00:00 2024-08-30T15:49:44.124 INFO:teuthology.orchestra.run.smithi050.stdout:(5/30): libradosstriper1-18.2.4-799.ge5753ca8.e 7.9 MB/s | 478 kB 00:00 2024-08-30T15:49:44.182 INFO:teuthology.orchestra.run.smithi050.stdout:(6/30): python3-ceph-argparse-18.2.4-799.ge5753 772 kB/s | 45 kB 00:00 2024-08-30T15:49:44.299 INFO:teuthology.orchestra.run.smithi050.stdout:(7/30): librgw2-18.2.4-799.ge5753ca8.el9.x86_64 22 MB/s | 4.5 MB 00:00 2024-08-30T15:49:44.325 INFO:teuthology.orchestra.run.smithi050.stdout:(8/30): python3-ceph-common-18.2.4-799.ge5753ca 911 kB/s | 129 kB 00:00 2024-08-30T15:49:44.358 INFO:teuthology.orchestra.run.smithi050.stdout:(9/30): python3-cephfs-18.2.4-799.ge5753ca8.el9 2.7 MB/s | 162 kB 00:00 2024-08-30T15:49:44.384 INFO:teuthology.orchestra.run.smithi050.stdout:(10/30): python3-rados-18.2.4-799.ge5753ca8.el9 5.3 MB/s | 320 kB 00:00 2024-08-30T15:49:44.409 INFO:teuthology.orchestra.run.smithi050.stdout:(11/30): python3-rbd-18.2.4-799.ge5753ca8.el9.x 5.8 MB/s | 298 kB 00:00 2024-08-30T15:49:44.434 INFO:teuthology.orchestra.run.smithi050.stdout:(12/30): python3-rgw-18.2.4-799.ge5753ca8.el9.x 1.9 MB/s | 99 kB 00:00 2024-08-30T15:49:44.618 INFO:teuthology.orchestra.run.smithi050.stdout:(13/30): ceph-common-18.2.4-799.ge5753ca8.el9.x 15 MB/s | 18 MB 00:01 2024-08-30T15:49:44.636 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:49:44.676 INFO:teuthology.orchestra.run.smithi050.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 400 kB/s | 107 kB 00:00 2024-08-30T15:49:44.710 INFO:teuthology.orchestra.run.smithi050.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 494 kB/s | 45 kB 00:00 2024-08-30T15:49:44.725 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-08-30T15:49:44.752 INFO:teuthology.orchestra.run.smithi050.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 504 kB/s | 160 kB 00:00 2024-08-30T15:49:44.754 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-08-30T15:49:44.777 INFO:teuthology.orchestra.run.smithi050.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-08-30T15:49:44.803 INFO:teuthology.orchestra.run.smithi050.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 837 kB/s | 42 kB 00:00 2024-08-30T15:49:44.803 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-08-30T15:49:44.828 INFO:teuthology.orchestra.run.smithi050.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-08-30T15:49:44.887 INFO:teuthology.orchestra.run.smithi050.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 1.0 MB/s | 111 kB 00:00 2024-08-30T15:49:44.912 INFO:teuthology.orchestra.run.smithi050.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-08-30T15:49:44.938 INFO:teuthology.orchestra.run.smithi050.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.2 MB/s | 309 kB 00:00 2024-08-30T15:49:44.963 INFO:teuthology.orchestra.run.smithi050.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-08-30T15:49:44.988 INFO:teuthology.orchestra.run.smithi050.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-08-30T15:49:45.030 INFO:teuthology.orchestra.run.smithi050.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-08-30T15:49:45.064 INFO:teuthology.orchestra.run.smithi050.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-08-30T15:49:45.117 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-08-30T15:49:45.205 INFO:teuthology.orchestra.run.smithi050.stdout:(27/30): librados2-18.2.4-799.ge5753ca8.el9.x86 23 MB/s | 3.3 MB 00:00 2024-08-30T15:49:45.229 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:18.2.4-799.ge5753ca8.el9.x86_64 5/32 2024-08-30T15:49:45.247 INFO:teuthology.orchestra.run.smithi050.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-08-30T15:49:45.320 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:18.2.4-799.ge5753ca8.el9.x86_64 5/32 2024-08-30T15:49:45.339 INFO:teuthology.orchestra.run.smithi050.stdout:(29/30): librbd1-18.2.4-799.ge5753ca8.el9.x86_6 23 MB/s | 3.0 MB 00:00 2024-08-30T15:49:45.404 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:18.2.4-799.ge5753ca8.el9.x86_64 6/32 2024-08-30T15:49:45.406 INFO:teuthology.orchestra.run.smithi050.stdout:(30/30): libarrow-9.0.0-11.el9.x86_64.rpm 7.9 MB/s | 4.4 MB 00:00 2024-08-30T15:49:45.410 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:49:45.411 INFO:teuthology.orchestra.run.smithi050.stdout:Total 23 MB/s | 53 MB 00:02 2024-08-30T15:49:45.430 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:18.2.4-799.ge5753ca8.el9.x86_64 7/32 2024-08-30T15:49:45.484 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:49:45.492 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:18.2.4-799.ge5753ca8.el9.x86_64 7/32 2024-08-30T15:49:45.521 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-08-30T15:49:45.522 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:49:45.522 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:49:45.590 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:18.2.4-799.ge5753ca8.el9.x 9/32 2024-08-30T15:49:45.656 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:18.2.4-799.ge5753ca8.el9.x86_64 10/32 2024-08-30T15:49:45.682 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:18.2.4-799.ge5753ca8.el9.x86_64 11/32 2024-08-30T15:49:45.740 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:18.2.4-799.ge5753ca8.el9.x86_64 11/32 2024-08-30T15:49:45.846 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-08-30T15:49:45.881 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-08-30T15:49:45.920 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-08-30T15:49:45.946 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-08-30T15:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:49:45.973 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-08-30T15:49:46.009 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-08-30T15:49:46.042 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-08-30T15:49:46.333 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-08-30T15:49:46.372 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:18.2.4-799.ge5753ca8.el9.x86_64 20/32 2024-08-30T15:49:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:18.2.4-799.ge5753ca8.el9.x86_64 20/32 2024-08-30T15:49:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:18.2.4-799.ge5753ca8.el9.x86_64 21/32 2024-08-30T15:49:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-08-30T15:49:46.933 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-08-30T15:49:46.957 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:18.2.4-799.ge5753ca8.el9.x86_64 24/32 2024-08-30T15:49:47.028 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:18.2.4-799.ge5753ca8.el9.x86_64 24/32 2024-08-30T15:49:47.105 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:18.2.4-799.ge5753ca8.el9.x86_64 25/32 2024-08-30T15:49:47.135 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:18.2.4-799.ge5753ca8.el9.x86 26/32 2024-08-30T15:49:47.501 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:49:47.591 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-08-30T15:49:47.628 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-08-30T15:49:47.695 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-08-30T15:49:48.073 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-08-30T15:49:48.152 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:18.2.4-799.ge5753ca8.el9.x86_64 5/32 2024-08-30T15:49:48.245 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:18.2.4-799.ge5753ca8.el9.x86_64 5/32 2024-08-30T15:49:48.332 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:18.2.4-799.ge5753ca8.el9.x86_64 6/32 2024-08-30T15:49:48.354 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:18.2.4-799.ge5753ca8.el9.x86_64 7/32 2024-08-30T15:49:48.399 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:18.2.4-799.ge5753ca8.el9.x86_64 7/32 2024-08-30T15:49:48.429 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-08-30T15:49:48.473 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:18.2.4-799.ge5753ca8.el9.x 9/32 2024-08-30T15:49:48.541 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:18.2.4-799.ge5753ca8.el9.x86_64 10/32 2024-08-30T15:49:48.564 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:18.2.4-799.ge5753ca8.el9.x86_64 11/32 2024-08-30T15:49:48.630 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:18.2.4-799.ge5753ca8.el9.x86_64 11/32 2024-08-30T15:49:48.740 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-08-30T15:49:48.772 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-08-30T15:49:48.811 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-08-30T15:49:48.845 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-08-30T15:49:48.872 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-08-30T15:49:48.909 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-08-30T15:49:48.941 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-08-30T15:49:49.257 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-08-30T15:49:49.320 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:18.2.4-799.ge5753ca8.el9.x86_64 27/32 2024-08-30T15:49:49.320 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:18.2.4-799.ge5753ca8.el9.x86_64 20/32 2024-08-30T15:49:49.352 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:18.2.4-799.ge5753ca8.el9.x86_64 27/32 2024-08-30T15:49:49.378 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:18.2.4-799.ge5753ca8.el9.x86_64 20/32 2024-08-30T15:49:49.409 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:18.2.4-799.ge5753ca8.el9.x86_64 21/32 2024-08-30T15:49:49.445 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-08-30T15:49:49.871 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:18.2.4-799.ge5753ca8.el9.x86_64 27/32 2024-08-30T15:49:49.871 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-08-30T15:49:49.899 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:18.2.4-799.ge5753ca8.el9.x86_64 24/32 2024-08-30T15:49:49.919 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:18.2.4-799.ge5753ca8.el9.x86_64 28/32 2024-08-30T15:49:49.961 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:18.2.4-799.ge5753ca8.el9.x86_64 24/32 2024-08-30T15:49:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:18.2.4-799.ge5753ca8.el9.x86_64 28/32 2024-08-30T15:49:50.019 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-30T15:49:50.019 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-30T15:49:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:49:50.032 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:18.2.4-799.ge5753ca8.el9.x86_64 25/32 2024-08-30T15:49:50.054 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:18.2.4-799.ge5753ca8.el9.x86_64 29/32 2024-08-30T15:49:50.075 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:18.2.4-799.ge5753ca8.el9.x86 26/32 2024-08-30T15:49:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:18.2.4-799.ge5753ca8.el9.x86_64 27/32 2024-08-30T15:49:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:18.2.4-799.ge5753ca8.el9.x86_64 27/32 2024-08-30T15:49:52.954 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:18.2.4-799.ge5753ca8.el9.x86_64 27/32 2024-08-30T15:49:53.129 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:18.2.4-799.ge5753ca8.el9.x86_64 28/32 2024-08-30T15:49:53.225 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:18.2.4-799.ge5753ca8.el9.x86_64 28/32 2024-08-30T15:49:53.226 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-30T15:49:53.226 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-30T15:49:53.226 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:49:53.255 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:18.2.4-799.ge5753ca8.el9.x86_64 29/32 2024-08-30T15:49:59.997 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:18.2.4-799.ge5753ca8.el9.x86_64 29/32 2024-08-30T15:49:59.998 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2024-08-30T15:49:59.998 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2024-08-30T15:49:59.998 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2024-08-30T15:49:59.998 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2024-08-30T15:49:59.998 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2024-08-30T15:49:59.998 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /root 2024-08-30T15:49:59.998 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2024-08-30T15:49:59.998 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:00.754 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:18.2.4-799.ge5753ca8.el9.x86_64 30/32 2024-08-30T15:50:00.779 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-799.ge5753ca8.el9.x86_64 30/32 2024-08-30T15:50:00.780 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:50:00.780 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-30T15:50:00.780 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-30T15:50:00.780 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-30T15:50:00.780 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:00.781 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-08-30T15:50:00.865 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-08-30T15:50:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-08-30T15:50:02.047 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-08-30T15:50:02.047 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:18.2.4-799.ge5753ca8.el9.x86_64 1/32 2024-08-30T15:50:02.047 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:18.2.4-799.ge5753ca8.el9.x86_64 2/32 2024-08-30T15:50:02.047 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:18.2.4-799.ge5753ca8.el9.x86_64 3/32 2024-08-30T15:50:02.047 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:18.2.4-799.ge5753ca8.el9.x86_64 4/32 2024-08-30T15:50:02.047 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:18.2.4-799.ge5753ca8.el9.x86_64 5/32 2024-08-30T15:50:02.047 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:18.2.4-799.ge5753ca8.el9.x86_64 6/32 2024-08-30T15:50:02.048 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:18.2.4-799.ge5753ca8.el9.x86_64 7/32 2024-08-30T15:50:02.048 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:18.2.4-799.ge5753ca8.el9.x 8/32 2024-08-30T15:50:02.048 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:18.2.4-799.ge5753ca8.el9.x86 9/32 2024-08-30T15:50:02.048 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:18.2.4-799.ge5753ca8.el9.x86_64 10/32 2024-08-30T15:50:02.048 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:18.2.4-799.ge5753ca8.el9.x86_64 11/32 2024-08-30T15:50:02.048 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:18.2.4-799.ge5753ca8.el9.x86_64 12/32 2024-08-30T15:50:02.048 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:18.2.4-799.ge5753ca8.el9.x86_64 13/32 2024-08-30T15:50:02.048 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-08-30T15:50:02.048 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-08-30T15:50:02.048 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-08-30T15:50:02.048 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-08-30T15:50:02.048 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-08-30T15:50:02.048 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-08-30T15:50:02.049 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-08-30T15:50:02.049 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-08-30T15:50:02.049 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-08-30T15:50:02.049 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-08-30T15:50:02.049 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-08-30T15:50:02.049 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-08-30T15:50:02.049 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-08-30T15:50:02.049 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-08-30T15:50:02.049 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-08-30T15:50:02.049 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:18.2.4-799.ge5753ca8.el9.x86_64 29/32 2024-08-30T15:50:02.049 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-08-30T15:50:02.050 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:18.2.4-799.ge5753ca8.el9.x86_64 31/32 2024-08-30T15:50:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-08-30T15:50:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:02.795 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2024-08-30T15:50:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.795 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:50:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-30T15:50:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.795 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: fmt-8.1.1-5.el9.x86_64 2024-08-30T15:50:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-30T15:50:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-30T15:50:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-30T15:50:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-30T15:50:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-30T15:50:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-30T15:50:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.796 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-30T15:50:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-30T15:50:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-30T15:50:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-30T15:50:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-30T15:50:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.797 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: re2-1:20211101-20.el9.x86_64 2024-08-30T15:50:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-30T15:50:02.798 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:02.798 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:50:03.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-test 2024-08-30T15:50:03.673 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:26 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:50:03.807 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:50:03.807 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:03.807 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-08-30T15:50:03.808 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:03.808 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:50:03.808 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 40 M 2024-08-30T15:50:03.808 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-08-30T15:50:03.808 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 166 k 2024-08-30T15:50:03.808 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-30T15:50:03.808 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-30T15:50:03.808 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:03.808 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:50:03.808 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:03.809 INFO:teuthology.orchestra.run.smithi094.stdout:Install 4 Packages 2024-08-30T15:50:03.809 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:03.809 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 40 M 2024-08-30T15:50:03.809 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 164 M 2024-08-30T15:50:03.810 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:50:03.857 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:18.2.4-799.ge5753ca8.el9.x86_64 29/32 2024-08-30T15:50:03.857 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2024-08-30T15:50:03.857 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2024-08-30T15:50:03.857 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2024-08-30T15:50:03.857 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2024-08-30T15:50:03.857 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2024-08-30T15:50:03.858 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /root 2024-08-30T15:50:03.858 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2024-08-30T15:50:03.858 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:04.050 INFO:teuthology.orchestra.run.smithi094.stdout:(1/4): libcephsqlite-18.2.4-799.ge5753ca8.el9.x 765 kB/s | 166 kB 00:00 2024-08-30T15:50:04.622 INFO:teuthology.orchestra.run.smithi094.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 385 kB/s | 304 kB 00:00 2024-08-30T15:50:04.655 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:18.2.4-799.ge5753ca8.el9.x86_64 30/32 2024-08-30T15:50:04.680 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-799.ge5753ca8.el9.x86_64 30/32 2024-08-30T15:50:04.680 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:50:04.680 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-30T15:50:04.680 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-30T15:50:04.680 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-30T15:50:04.680 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:04.682 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-08-30T15:50:04.723 INFO:teuthology.orchestra.run.smithi094.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 95 kB/s | 64 kB 00:00 2024-08-30T15:50:04.766 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-08-30T15:50:04.766 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-08-30T15:50:05.125 INFO:teuthology.orchestra.run.smithi094.stdout:(4/4): ceph-test-18.2.4-799.ge5753ca8.el9.x86_6 31 MB/s | 40 MB 00:01 2024-08-30T15:50:05.128 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:05.128 INFO:teuthology.orchestra.run.smithi094.stdout:Total 30 MB/s | 40 MB 00:01 2024-08-30T15:50:05.157 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:50:05.179 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:50:05.179 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:50:05.514 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:50:05.515 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:50:05.985 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-08-30T15:50:05.985 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:18.2.4-799.ge5753ca8.el9.x86_64 1/32 2024-08-30T15:50:05.985 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:18.2.4-799.ge5753ca8.el9.x86_64 2/32 2024-08-30T15:50:05.985 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:18.2.4-799.ge5753ca8.el9.x86_64 3/32 2024-08-30T15:50:05.986 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:18.2.4-799.ge5753ca8.el9.x86_64 4/32 2024-08-30T15:50:05.986 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:18.2.4-799.ge5753ca8.el9.x86_64 5/32 2024-08-30T15:50:05.986 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:18.2.4-799.ge5753ca8.el9.x86_64 6/32 2024-08-30T15:50:05.986 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:18.2.4-799.ge5753ca8.el9.x86_64 7/32 2024-08-30T15:50:05.986 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:18.2.4-799.ge5753ca8.el9.x 8/32 2024-08-30T15:50:05.986 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:18.2.4-799.ge5753ca8.el9.x86 9/32 2024-08-30T15:50:05.986 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:18.2.4-799.ge5753ca8.el9.x86_64 10/32 2024-08-30T15:50:05.986 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:18.2.4-799.ge5753ca8.el9.x86_64 11/32 2024-08-30T15:50:05.986 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:18.2.4-799.ge5753ca8.el9.x86_64 12/32 2024-08-30T15:50:05.986 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:18.2.4-799.ge5753ca8.el9.x86_64 13/32 2024-08-30T15:50:05.986 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-08-30T15:50:05.986 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-08-30T15:50:05.986 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-08-30T15:50:05.987 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-08-30T15:50:05.987 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-08-30T15:50:05.987 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-08-30T15:50:05.987 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-08-30T15:50:05.987 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-08-30T15:50:05.987 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-08-30T15:50:05.987 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-08-30T15:50:05.987 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-08-30T15:50:05.987 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-08-30T15:50:05.987 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-08-30T15:50:05.988 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-08-30T15:50:05.988 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-08-30T15:50:05.988 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:18.2.4-799.ge5753ca8.el9.x86_64 29/32 2024-08-30T15:50:05.988 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-08-30T15:50:05.988 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:18.2.4-799.ge5753ca8.el9.x86_64 31/32 2024-08-30T15:50:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:50:06.181 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-30T15:50:06.222 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-30T15:50:06.249 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:18.2.4-799.ge5753ca8.el9.x86_64 3/4 2024-08-30T15:50:06.753 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-08-30T15:50:06.753 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:06.753 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2024-08-30T15:50:06.753 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.753 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.753 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:50:06.753 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-30T15:50:06.753 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.754 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.754 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.754 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.754 INFO:teuthology.orchestra.run.smithi050.stdout: fmt-8.1.1-5.el9.x86_64 2024-08-30T15:50:06.754 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-30T15:50:06.754 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-30T15:50:06.754 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-30T15:50:06.754 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.754 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-30T15:50:06.754 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-30T15:50:06.754 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-30T15:50:06.754 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.755 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-30T15:50:06.755 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.755 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-30T15:50:06.755 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-30T15:50:06.755 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-30T15:50:06.755 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.755 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.755 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.755 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-30T15:50:06.755 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.755 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.755 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:06.756 INFO:teuthology.orchestra.run.smithi050.stdout: re2-1:20211101-20.el9.x86_64 2024-08-30T15:50:06.756 INFO:teuthology.orchestra.run.smithi050.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-30T15:50:06.756 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:06.756 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:50:07.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test 2024-08-30T15:50:07.683 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:27 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:50:07.834 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:50:07.835 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:07.835 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-08-30T15:50:07.835 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:07.835 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:50:07.835 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 40 M 2024-08-30T15:50:07.836 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-08-30T15:50:07.836 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 166 k 2024-08-30T15:50:07.836 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-30T15:50:07.836 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-30T15:50:07.836 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:07.836 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:50:07.836 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:07.836 INFO:teuthology.orchestra.run.smithi050.stdout:Install 4 Packages 2024-08-30T15:50:07.836 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:07.837 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 40 M 2024-08-30T15:50:07.837 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 164 M 2024-08-30T15:50:07.837 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:50:08.319 INFO:teuthology.orchestra.run.smithi050.stdout:(1/4): libcephsqlite-18.2.4-799.ge5753ca8.el9.x 744 kB/s | 166 kB 00:00 2024-08-30T15:50:08.672 INFO:teuthology.orchestra.run.smithi050.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 180 kB/s | 64 kB 00:00 2024-08-30T15:50:08.731 INFO:teuthology.orchestra.run.smithi050.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 479 kB/s | 304 kB 00:00 2024-08-30T15:50:09.377 INFO:teuthology.orchestra.run.smithi050.stdout:(4/4): ceph-test-18.2.4-799.ge5753ca8.el9.x86_6 31 MB/s | 40 MB 00:01 2024-08-30T15:50:09.379 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:09.379 INFO:teuthology.orchestra.run.smithi050.stdout:Total 26 MB/s | 40 MB 00:01 2024-08-30T15:50:09.406 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:50:09.428 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:50:09.428 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:50:09.759 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:50:09.759 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:50:09.895 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:18.2.4-799.ge5753ca8.el9.x86_64 3/4 2024-08-30T15:50:09.963 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:18.2.4-799.ge5753ca8.el9.x86_64 4/4 2024-08-30T15:50:10.239 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:50:10.322 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-30T15:50:10.362 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-30T15:50:10.398 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:18.2.4-799.ge5753ca8.el9.x86_64 3/4 2024-08-30T15:50:12.000 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-test-2:18.2.4-799.ge5753ca8.el9.x86_64 4/4 2024-08-30T15:50:12.000 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:18.2.4-799.ge5753ca8.el9.x86_64 1/4 2024-08-30T15:50:12.000 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:18.2.4-799.ge5753ca8.el9.x86_64 2/4 2024-08-30T15:50:12.000 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-30T15:50:12.282 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-30T15:50:12.282 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:12.282 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:50:12.282 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:12.282 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:12.282 INFO:teuthology.orchestra.run.smithi094.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-30T15:50:12.282 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-30T15:50:12.282 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:12.282 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:50:12.583 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph 2024-08-30T15:50:13.143 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:36 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:50:13.280 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:50:13.283 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-08-30T15:50:13.283 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:13.283 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:50:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 6.5 k 2024-08-30T15:50:13.283 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-08-30T15:50:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 2.1 M 2024-08-30T15:50:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 1.5 M 2024-08-30T15:50:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 246 k 2024-08-30T15:50:13.283 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 4.7 M 2024-08-30T15:50:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 17 M 2024-08-30T15:50:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-30T15:50:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-30T15:50:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-30T15:50:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-30T15:50:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-30T15:50:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-30T15:50:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-30T15:50:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-30T15:50:13.284 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-30T15:50:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-30T15:50:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-30T15:50:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-30T15:50:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-30T15:50:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-30T15:50:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-30T15:50:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-30T15:50:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-30T15:50:13.285 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-30T15:50:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-30T15:50:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-30T15:50:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-30T15:50:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-30T15:50:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-30T15:50:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-30T15:50:13.286 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-08-30T15:50:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-30T15:50:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-30T15:50:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:13.287 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:50:13.287 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:13.287 INFO:teuthology.orchestra.run.smithi094.stdout:Install 33 Packages 2024-08-30T15:50:13.287 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:13.288 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 29 M 2024-08-30T15:50:13.288 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 104 M 2024-08-30T15:50:13.288 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:50:13.766 INFO:teuthology.orchestra.run.smithi094.stdout:(1/33): ceph-18.2.4-799.ge5753ca8.el9.x86_64.rp 44 kB/s | 6.5 kB 00:00 2024-08-30T15:50:13.933 INFO:teuthology.orchestra.run.smithi094.stdout:(2/33): ceph-mgr-18.2.4-799.ge5753ca8.el9.x86_6 4.7 MB/s | 1.5 MB 00:00 2024-08-30T15:50:13.975 INFO:teuthology.orchestra.run.smithi094.stdout:(3/33): ceph-mds-18.2.4-799.ge5753ca8.el9.x86_6 5.9 MB/s | 2.1 MB 00:00 2024-08-30T15:50:14.042 INFO:teuthology.orchestra.run.smithi094.stdout:(4/33): ceph-mgr-modules-core-18.2.4-799.ge5753 3.6 MB/s | 246 kB 00:00 2024-08-30T15:50:14.176 INFO:teuthology.orchestra.run.smithi094.stdout:(5/33): ceph-mon-18.2.4-799.ge5753ca8.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-08-30T15:50:14.265 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:18.2.4-799.ge5753ca8.el9.x86_64 3/4 2024-08-30T15:50:14.370 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:18.2.4-799.ge5753ca8.el9.x86_64 4/4 2024-08-30T15:50:14.509 INFO:teuthology.orchestra.run.smithi094.stdout:(6/33): ceph-osd-18.2.4-799.ge5753ca8.el9.x86_6 29 MB/s | 17 MB 00:00 2024-08-30T15:50:14.984 INFO:teuthology.orchestra.run.smithi094.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 269 kB/s | 253 kB 00:00 2024-08-30T15:50:15.202 INFO:teuthology.orchestra.run.smithi094.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 156 kB/s | 106 kB 00:00 2024-08-30T15:50:15.260 INFO:teuthology.orchestra.run.smithi094.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 1.1 MB/s | 1.2 MB 00:01 2024-08-30T15:50:15.694 INFO:teuthology.orchestra.run.smithi094.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 191 kB/s | 135 kB 00:00 2024-08-30T15:50:15.894 INFO:teuthology.orchestra.run.smithi094.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 861 kB/s | 172 kB 00:00 2024-08-30T15:50:15.919 INFO:teuthology.orchestra.run.smithi094.stdout:(12/33): python3-requests-2.25.1-8.el9.noarch.r 176 kB/s | 125 kB 00:00 2024-08-30T15:50:15.945 INFO:teuthology.orchestra.run.smithi094.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 706 kB/s | 35 kB 00:00 2024-08-30T15:50:15.995 INFO:teuthology.orchestra.run.smithi094.stdout:(14/33): python3-toml-0.10.2-6.el9.noarch.rpm 835 kB/s | 42 kB 00:00 2024-08-30T15:50:16.020 INFO:teuthology.orchestra.run.smithi094.stdout:(15/33): python3-urllib3-1.26.5-6.el9.noarch.rp 283 kB/s | 215 kB 00:00 2024-08-30T15:50:16.062 INFO:teuthology.orchestra.run.smithi094.stdout:(16/33): python3-pytz-2021.1-5.el9.noarch.rpm 358 kB/s | 51 kB 00:00 2024-08-30T15:50:16.087 INFO:teuthology.orchestra.run.smithi094.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 477 kB/s | 43 kB 00:00 2024-08-30T15:50:16.116 INFO:teuthology.orchestra.run.smithi094.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 497 kB/s | 11 kB 00:00 2024-08-30T15:50:16.142 INFO:teuthology.orchestra.run.smithi094.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-08-30T15:50:16.167 INFO:teuthology.orchestra.run.smithi094.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 365 kB/s | 18 kB 00:00 2024-08-30T15:50:16.192 INFO:teuthology.orchestra.run.smithi094.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-08-30T15:50:16.217 INFO:teuthology.orchestra.run.smithi094.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-08-30T15:50:16.243 INFO:teuthology.orchestra.run.smithi094.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-08-30T15:50:16.268 INFO:teuthology.orchestra.run.smithi094.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-08-30T15:50:16.294 INFO:teuthology.orchestra.run.smithi094.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-08-30T15:50:16.319 INFO:teuthology.orchestra.run.smithi094.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-08-30T15:50:16.344 INFO:teuthology.orchestra.run.smithi094.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-08-30T15:50:16.378 INFO:teuthology.orchestra.run.smithi094.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-08-30T15:50:16.412 INFO:teuthology.orchestra.run.smithi094.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 975 kB/s | 90 kB 00:00 2024-08-30T15:50:16.434 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-2:18.2.4-799.ge5753ca8.el9.x86_64 4/4 2024-08-30T15:50:16.434 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:18.2.4-799.ge5753ca8.el9.x86_64 1/4 2024-08-30T15:50:16.434 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:18.2.4-799.ge5753ca8.el9.x86_64 2/4 2024-08-30T15:50:16.434 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-30T15:50:16.436 INFO:teuthology.orchestra.run.smithi094.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2024-08-30T15:50:16.462 INFO:teuthology.orchestra.run.smithi094.stdout:(31/33): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-08-30T15:50:16.487 INFO:teuthology.orchestra.run.smithi094.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-08-30T15:50:16.512 INFO:teuthology.orchestra.run.smithi094.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-08-30T15:50:16.518 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:16.518 INFO:teuthology.orchestra.run.smithi094.stdout:Total 9.0 MB/s | 29 MB 00:03 2024-08-30T15:50:16.640 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:50:16.678 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:50:16.678 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:50:16.750 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-30T15:50:16.750 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:16.750 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:50:16.751 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:16.751 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:16.751 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-30T15:50:16.751 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-30T15:50:16.751 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:16.751 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:50:16.959 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:50:16.960 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:50:17.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph 2024-08-30T15:50:17.587 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:37 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:50:17.725 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:50:17.727 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:17.728 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-08-30T15:50:17.728 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:17.728 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:50:17.728 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 6.5 k 2024-08-30T15:50:17.728 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-08-30T15:50:17.728 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 2.1 M 2024-08-30T15:50:17.728 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 1.5 M 2024-08-30T15:50:17.728 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 246 k 2024-08-30T15:50:17.728 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 4.7 M 2024-08-30T15:50:17.728 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 17 M 2024-08-30T15:50:17.728 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-30T15:50:17.729 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-30T15:50:17.729 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-30T15:50:17.729 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-30T15:50:17.729 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-30T15:50:17.729 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-30T15:50:17.729 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-30T15:50:17.729 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-30T15:50:17.729 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-30T15:50:17.729 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-30T15:50:17.729 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-30T15:50:17.729 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-30T15:50:17.729 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-30T15:50:17.730 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-30T15:50:17.730 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-30T15:50:17.730 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-30T15:50:17.730 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-30T15:50:17.730 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-30T15:50:17.730 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-30T15:50:17.730 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-30T15:50:17.730 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-30T15:50:17.730 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-30T15:50:17.730 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-30T15:50:17.730 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-30T15:50:17.730 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-08-30T15:50:17.730 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-30T15:50:17.730 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-30T15:50:17.731 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:17.731 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:50:17.731 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:17.731 INFO:teuthology.orchestra.run.smithi050.stdout:Install 33 Packages 2024-08-30T15:50:17.731 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:17.731 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 29 M 2024-08-30T15:50:17.732 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 104 M 2024-08-30T15:50:17.732 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:50:18.186 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:50:18.249 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-08-30T15:50:18.282 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-08-30T15:50:18.314 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-08-30T15:50:18.373 INFO:teuthology.orchestra.run.smithi050.stdout:(1/33): ceph-18.2.4-799.ge5753ca8.el9.x86_64.rp 47 kB/s | 6.5 kB 00:00 2024-08-30T15:50:18.406 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-08-30T15:50:18.493 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-08-30T15:50:18.533 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-08-30T15:50:18.548 INFO:teuthology.orchestra.run.smithi050.stdout:(2/33): ceph-mds-18.2.4-799.ge5753ca8.el9.x86_6 6.7 MB/s | 2.1 MB 00:00 2024-08-30T15:50:18.577 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-08-30T15:50:18.590 INFO:teuthology.orchestra.run.smithi050.stdout:(3/33): ceph-mgr-18.2.4-799.ge5753ca8.el9.x86_6 4.1 MB/s | 1.5 MB 00:00 2024-08-30T15:50:18.607 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-08-30T15:50:18.641 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-08-30T15:50:18.673 INFO:teuthology.orchestra.run.smithi050.stdout:(4/33): ceph-mgr-modules-core-18.2.4-799.ge5753 2.9 MB/s | 246 kB 00:00 2024-08-30T15:50:18.675 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-08-30T15:50:18.715 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-08-30T15:50:18.750 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-08-30T15:50:18.785 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-08-30T15:50:18.815 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-08-30T15:50:18.888 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-08-30T15:50:19.043 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-08-30T15:50:19.074 INFO:teuthology.orchestra.run.smithi050.stdout:(5/33): ceph-osd-18.2.4-799.ge5753ca8.el9.x86_6 31 MB/s | 17 MB 00:00 2024-08-30T15:50:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-08-30T15:50:19.141 INFO:teuthology.orchestra.run.smithi050.stdout:(6/33): ceph-mon-18.2.4-799.ge5753ca8.el9.x86_6 6.1 MB/s | 4.7 MB 00:00 2024-08-30T15:50:19.176 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-08-30T15:50:19.223 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-08-30T15:50:19.273 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-08-30T15:50:19.327 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-08-30T15:50:19.515 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-08-30T15:50:19.549 INFO:teuthology.orchestra.run.smithi050.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 289 kB/s | 253 kB 00:00 2024-08-30T15:50:19.561 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-08-30T15:50:19.639 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-08-30T15:50:19.736 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-08-30T15:50:19.777 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-08-30T15:50:19.816 INFO:teuthology.orchestra.run.smithi050.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 158 kB/s | 106 kB 00:00 2024-08-30T15:50:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-08-30T15:50:20.017 INFO:teuthology.orchestra.run.smithi050.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 1.3 MB/s | 1.2 MB 00:00 2024-08-30T15:50:20.061 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-799.ge5753ca8.el9.n 28/33 2024-08-30T15:50:20.087 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:18.2.4-799.ge5753ca8.el9.x86_64 29/33 2024-08-30T15:50:20.113 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:18.2.4-799.ge5753ca8.el9.x86_64 29/33 2024-08-30T15:50:20.113 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:50:20.113 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-30T15:50:20.113 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-30T15:50:20.114 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-30T15:50:20.114 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:20.276 INFO:teuthology.orchestra.run.smithi050.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 186 kB/s | 135 kB 00:00 2024-08-30T15:50:20.509 INFO:teuthology.orchestra.run.smithi050.stdout:(11/33): python3-requests-2.25.1-8.el9.noarch.r 180 kB/s | 125 kB 00:00 2024-08-30T15:50:20.759 INFO:teuthology.orchestra.run.smithi050.stdout:(12/33): python3-urllib3-1.26.5-6.el9.noarch.rp 289 kB/s | 215 kB 00:00 2024-08-30T15:50:21.018 INFO:teuthology.orchestra.run.smithi050.stdout:(13/33): python3-mako-1.1.4-6.el9.noarch.rpm 232 kB/s | 172 kB 00:00 2024-08-30T15:50:21.068 INFO:teuthology.orchestra.run.smithi050.stdout:(14/33): python3-markupsafe-1.1.1-12.el9.x86_64 63 kB/s | 35 kB 00:00 2024-08-30T15:50:21.118 INFO:teuthology.orchestra.run.smithi050.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 877 kB/s | 43 kB 00:00 2024-08-30T15:50:21.160 INFO:teuthology.orchestra.run.smithi050.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 4.1 MB/s | 172 kB 00:00 2024-08-30T15:50:21.211 INFO:teuthology.orchestra.run.smithi050.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 7.0 MB/s | 358 kB 00:00 2024-08-30T15:50:21.236 INFO:teuthology.orchestra.run.smithi050.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 432 kB/s | 11 kB 00:00 2024-08-30T15:50:21.261 INFO:teuthology.orchestra.run.smithi050.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 707 kB/s | 18 kB 00:00 2024-08-30T15:50:21.286 INFO:teuthology.orchestra.run.smithi050.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 925 kB/s | 23 kB 00:00 2024-08-30T15:50:21.320 INFO:teuthology.orchestra.run.smithi050.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 769 kB/s | 19 kB 00:00 2024-08-30T15:50:21.345 INFO:teuthology.orchestra.run.smithi050.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 793 kB/s | 20 kB 00:00 2024-08-30T15:50:21.370 INFO:teuthology.orchestra.run.smithi050.stdout:(23/33): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-08-30T15:50:21.404 INFO:teuthology.orchestra.run.smithi050.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 2.4 MB/s | 79 kB 00:00 2024-08-30T15:50:21.440 INFO:teuthology.orchestra.run.smithi050.stdout:(25/33): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-08-30T15:50:21.463 INFO:teuthology.orchestra.run.smithi050.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 656 kB/s | 16 kB 00:00 2024-08-30T15:50:21.488 INFO:teuthology.orchestra.run.smithi050.stdout:(27/33): python3-pytz-2021.1-5.el9.noarch.rpm 70 kB/s | 51 kB 00:00 2024-08-30T15:50:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:18.2.4-799.ge5753ca8.el9.x86_64 30/33 2024-08-30T15:50:21.522 INFO:teuthology.orchestra.run.smithi050.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-08-30T15:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:18.2.4-799.ge5753ca8.el9.x86_64 30/33 2024-08-30T15:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-30T15:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-30T15:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-30T15:50:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2024-08-30T15:50:21.597 INFO:teuthology.orchestra.run.smithi050.stdout:(30/33): python3-webob-1.8.8-2.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-08-30T15:50:21.614 INFO:teuthology.orchestra.run.smithi050.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-08-30T15:50:21.673 INFO:teuthology.orchestra.run.smithi050.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-08-30T15:50:21.690 INFO:teuthology.orchestra.run.smithi050.stdout:(33/33): python3-toml-0.10.2-6.el9.noarch.rpm 63 kB/s | 42 kB 00:00 2024-08-30T15:50:21.696 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:21.696 INFO:teuthology.orchestra.run.smithi050.stdout:Total 7.4 MB/s | 29 MB 00:03 2024-08-30T15:50:21.821 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:50:21.862 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:50:21.862 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:50:21.980 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:18.2.4-799.ge5753ca8.el9.x86_64 31/33 2024-08-30T15:50:22.005 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:18.2.4-799.ge5753ca8.el9.x86_64 31/33 2024-08-30T15:50:22.005 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:50:22.005 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-30T15:50:22.006 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-30T15:50:22.006 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-30T15:50:22.006 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:22.157 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:50:22.159 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:50:22.214 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:18.2.4-799.ge5753ca8.el9.x86_64 32/33 2024-08-30T15:50:22.238 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:18.2.4-799.ge5753ca8.el9.x86_64 32/33 2024-08-30T15:50:22.239 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:50:22.239 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-30T15:50:22.239 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-30T15:50:22.239 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-30T15:50:22.239 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:22.301 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:18.2.4-799.ge5753ca8.el9.x86_64 33/33 2024-08-30T15:50:23.611 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:50:23.676 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-08-30T15:50:23.709 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-08-30T15:50:23.742 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-08-30T15:50:23.847 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-08-30T15:50:23.932 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-08-30T15:50:23.979 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-08-30T15:50:23.982 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-2:18.2.4-799.ge5753ca8.el9.x86_64 33/33 2024-08-30T15:50:23.982 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:18.2.4-799.ge5753ca8.el9.x86_64 1/33 2024-08-30T15:50:23.982 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:18.2.4-799.ge5753ca8.el9.x86_64 2/33 2024-08-30T15:50:23.982 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:18.2.4-799.ge5753ca8.el9.x86_64 3/33 2024-08-30T15:50:23.982 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:18.2.4-799.ge5753ca8.el9.x86_64 4/33 2024-08-30T15:50:23.982 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:18.2.4-799.ge5753ca8.el9.x86_64 5/33 2024-08-30T15:50:23.982 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-799.ge5753ca8.el9.n 6/33 2024-08-30T15:50:23.982 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-08-30T15:50:23.982 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-08-30T15:50:23.982 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-08-30T15:50:23.982 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-08-30T15:50:23.983 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-08-30T15:50:23.983 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-08-30T15:50:23.983 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-08-30T15:50:23.983 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-08-30T15:50:23.983 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-08-30T15:50:23.983 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-08-30T15:50:23.983 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-08-30T15:50:23.983 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-08-30T15:50:23.983 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-08-30T15:50:23.983 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-08-30T15:50:23.983 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-08-30T15:50:23.983 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-08-30T15:50:23.983 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-08-30T15:50:23.984 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-08-30T15:50:23.984 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-08-30T15:50:23.985 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-08-30T15:50:23.985 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-08-30T15:50:23.985 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-08-30T15:50:23.985 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-08-30T15:50:23.985 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-08-30T15:50:23.985 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-08-30T15:50:23.985 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-08-30T15:50:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-08-30T15:50:24.051 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-08-30T15:50:24.086 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-08-30T15:50:24.119 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-08-30T15:50:24.161 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-08-30T15:50:24.204 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-08-30T15:50:24.237 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-08-30T15:50:24.276 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-08-30T15:50:24.352 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-08-30T15:50:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-08-30T15:50:24.645 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-08-30T15:50:24.697 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-08-30T15:50:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-08-30T15:50:24.793 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-08-30T15:50:24.865 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-08-30T15:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-08-30T15:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:24.899 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-30T15:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-30T15:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-30T15:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-30T15:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-30T15:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-30T15:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-30T15:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-30T15:50:24.900 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-30T15:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-30T15:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-30T15:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-30T15:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-30T15:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-30T15:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-30T15:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply-3.11-14.el9.noarch 2024-08-30T15:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-30T15:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-30T15:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-30T15:50:24.901 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-30T15:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-30T15:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-30T15:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-30T15:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-30T15:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.8-2.el9.noarch 2024-08-30T15:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-30T15:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-30T15:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:24.902 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:50:25.073 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-08-30T15:50:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-08-30T15:50:25.202 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-08-30T15:50:25.254 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-base 2024-08-30T15:50:25.301 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-08-30T15:50:25.346 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-08-30T15:50:25.472 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-08-30T15:50:25.628 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-799.ge5753ca8.el9.n 28/33 2024-08-30T15:50:25.656 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:18.2.4-799.ge5753ca8.el9.x86_64 29/33 2024-08-30T15:50:25.681 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:18.2.4-799.ge5753ca8.el9.x86_64 29/33 2024-08-30T15:50:25.681 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:50:25.682 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-30T15:50:25.682 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-30T15:50:25.682 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-30T15:50:25.682 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:25.813 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:48 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:50:25.901 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-base-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:50:25.942 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:50:25.943 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-08-30T15:50:25.943 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:50:26.009 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install cephadm 2024-08-30T15:50:26.553 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:49 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:50:26.683 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:50:26.683 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:26.683 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-08-30T15:50:26.683 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:26.684 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:50:26.684 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 226 k 2024-08-30T15:50:26.684 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:26.684 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:50:26.684 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:26.684 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:50:26.684 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:26.684 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 226 k 2024-08-30T15:50:26.684 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 217 k 2024-08-30T15:50:26.684 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:50:27.185 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:18.2.4-799.ge5753ca8.el9.x86_64 30/33 2024-08-30T15:50:27.210 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:18.2.4-799.ge5753ca8.el9.x86_64 30/33 2024-08-30T15:50:27.211 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:50:27.211 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-30T15:50:27.211 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-30T15:50:27.211 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-30T15:50:27.211 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:27.306 INFO:teuthology.orchestra.run.smithi094.stdout:cephadm-18.2.4-799.ge5753ca8.el9.noarch.rpm 364 kB/s | 226 kB 00:00 2024-08-30T15:50:27.306 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:27.307 INFO:teuthology.orchestra.run.smithi094.stdout:Total 363 kB/s | 226 kB 00:00 2024-08-30T15:50:27.307 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:50:27.312 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:50:27.312 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:50:27.314 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:50:27.315 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:50:27.389 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:50:27.683 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:18.2.4-799.ge5753ca8.el9.x86_64 31/33 2024-08-30T15:50:27.707 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:18.2.4-799.ge5753ca8.el9.x86_64 31/33 2024-08-30T15:50:27.707 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:50:27.707 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-30T15:50:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-30T15:50:27.708 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-30T15:50:27.708 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:27.843 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 1/1 2024-08-30T15:50:27.924 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 1/1 2024-08-30T15:50:27.926 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:18.2.4-799.ge5753ca8.el9.x86_64 32/33 2024-08-30T15:50:27.949 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:18.2.4-799.ge5753ca8.el9.x86_64 32/33 2024-08-30T15:50:27.949 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:50:27.949 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-30T15:50:27.949 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-30T15:50:27.950 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-30T15:50:27.950 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:28.015 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:18.2.4-799.ge5753ca8.el9.x86_64 33/33 2024-08-30T15:50:28.257 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 1/1 2024-08-30T15:50:28.503 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 1/1 2024-08-30T15:50:28.503 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:28.503 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:50:28.504 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:50:28.504 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:28.504 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:50:28.703 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-immutable-object-cache 2024-08-30T15:50:29.263 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:52 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:50:29.395 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:50:29.395 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:29.396 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:50:29.396 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:29.396 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:50:29.396 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 143 k 2024-08-30T15:50:29.396 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:29.396 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:50:29.396 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:29.396 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:50:29.396 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:29.396 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 143 k 2024-08-30T15:50:29.396 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 395 k 2024-08-30T15:50:29.396 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:50:29.638 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-immutable-object-cache-18.2.4-799.ge5753ca 593 kB/s | 143 kB 00:00 2024-08-30T15:50:29.639 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:29.639 INFO:teuthology.orchestra.run.smithi094.stdout:Total 588 kB/s | 143 kB 00:00 2024-08-30T15:50:29.639 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:50:29.645 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:50:29.646 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:50:29.702 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:50:29.703 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:50:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-2:18.2.4-799.ge5753ca8.el9.x86_64 33/33 2024-08-30T15:50:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:18.2.4-799.ge5753ca8.el9.x86_64 1/33 2024-08-30T15:50:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:18.2.4-799.ge5753ca8.el9.x86_64 2/33 2024-08-30T15:50:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:18.2.4-799.ge5753ca8.el9.x86_64 3/33 2024-08-30T15:50:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:18.2.4-799.ge5753ca8.el9.x86_64 4/33 2024-08-30T15:50:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:18.2.4-799.ge5753ca8.el9.x86_64 5/33 2024-08-30T15:50:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-799.ge5753ca8.el9.n 6/33 2024-08-30T15:50:29.748 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-08-30T15:50:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-08-30T15:50:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-08-30T15:50:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-08-30T15:50:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-08-30T15:50:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-08-30T15:50:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-08-30T15:50:29.749 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-08-30T15:50:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-08-30T15:50:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-08-30T15:50:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-08-30T15:50:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-08-30T15:50:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-08-30T15:50:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-08-30T15:50:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-08-30T15:50:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-08-30T15:50:29.750 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-08-30T15:50:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-08-30T15:50:29.751 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-08-30T15:50:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-08-30T15:50:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-08-30T15:50:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-08-30T15:50:29.752 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-08-30T15:50:29.753 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-08-30T15:50:29.753 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-08-30T15:50:29.753 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-08-30T15:50:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:50:29.938 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-799.ge5753ca8.e 1/1 2024-08-30T15:50:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-799.ge5753ca8.e 1/1 2024-08-30T15:50:29.963 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:50:29.963 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-30T15:50:29.963 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-08-30T15:50:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:30.664 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:50:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:50:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-30T15:50:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-30T15:50:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-30T15:50:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-30T15:50:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-30T15:50:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-30T15:50:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-30T15:50:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-30T15:50:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-30T15:50:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-30T15:50:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-30T15:50:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-30T15:50:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-30T15:50:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-30T15:50:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-30T15:50:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply-3.11-14.el9.noarch 2024-08-30T15:50:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-30T15:50:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-30T15:50:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-30T15:50:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-30T15:50:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-30T15:50:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-30T15:50:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-30T15:50:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-30T15:50:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.8-2.el9.noarch 2024-08-30T15:50:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-30T15:50:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-30T15:50:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:30.667 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:50:30.878 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-799.ge5753ca8.e 1/1 2024-08-30T15:50:30.878 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:30.878 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:50:30.878 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:30.878 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:30.878 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:50:31.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base 2024-08-30T15:50:31.093 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr 2024-08-30T15:50:31.585 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:51 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:50:31.652 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:54 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:50:31.678 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:50:31.721 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:50:31.722 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-08-30T15:50:31.722 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:50:31.744 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-mgr-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:50:31.788 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:50:31.789 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-08-30T15:50:31.789 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:50:31.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install cephadm 2024-08-30T15:50:31.847 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-dashboard 2024-08-30T15:50:32.356 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:52 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:50:32.396 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:55 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:50:32.491 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:50:32.491 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:32.492 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-08-30T15:50:32.492 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:32.492 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:50:32.492 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 226 k 2024-08-30T15:50:32.492 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:32.492 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:50:32.492 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:32.492 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:50:32.492 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:32.492 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 226 k 2024-08-30T15:50:32.492 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 217 k 2024-08-30T15:50:32.492 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:50:32.525 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:50:32.526 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-08-30T15:50:32.526 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:32.526 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:50:32.526 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 3.5 M 2024-08-30T15:50:32.527 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-08-30T15:50:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 24 k 2024-08-30T15:50:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 16 k 2024-08-30T15:50:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-30T15:50:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-30T15:50:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:32.527 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:50:32.527 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:32.527 INFO:teuthology.orchestra.run.smithi094.stdout:Install 5 Packages 2024-08-30T15:50:32.527 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:32.528 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 3.8 M 2024-08-30T15:50:32.528 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 87 M 2024-08-30T15:50:32.528 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:50:32.707 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm-18.2.4-799.ge5753ca8.el9.noarch.rpm 1.0 MB/s | 226 kB 00:00 2024-08-30T15:50:32.708 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:32.708 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.0 MB/s | 226 kB 00:00 2024-08-30T15:50:32.708 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:50:32.714 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:50:32.714 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:50:32.716 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:50:32.717 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:50:32.737 INFO:teuthology.orchestra.run.smithi094.stdout:(1/5): ceph-grafana-dashboards-18.2.4-799.ge575 167 kB/s | 24 kB 00:00 2024-08-30T15:50:32.762 INFO:teuthology.orchestra.run.smithi094.stdout:(2/5): ceph-prometheus-alerts-18.2.4-799.ge5753 96 kB/s | 16 kB 00:00 2024-08-30T15:50:32.787 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:50:32.804 INFO:teuthology.orchestra.run.smithi094.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 459 kB/s | 31 kB 00:00 2024-08-30T15:50:32.838 INFO:teuthology.orchestra.run.smithi094.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-08-30T15:50:33.004 INFO:teuthology.orchestra.run.smithi094.stdout:(5/5): ceph-mgr-dashboard-18.2.4-799.ge5753ca8. 8.6 MB/s | 3.5 MB 00:00 2024-08-30T15:50:33.008 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:33.008 INFO:teuthology.orchestra.run.smithi094.stdout:Total 7.9 MB/s | 3.8 MB 00:00 2024-08-30T15:50:33.008 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:50:33.015 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:50:33.016 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:50:33.078 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:50:33.078 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:50:33.170 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 1/1 2024-08-30T15:50:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 1/1 2024-08-30T15:50:33.326 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:50:33.403 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-08-30T15:50:33.435 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-08-30T15:50:33.463 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-799.ge5753ca8.el9.no 3/5 2024-08-30T15:50:33.646 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 1/1 2024-08-30T15:50:33.884 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 1/1 2024-08-30T15:50:33.885 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:33.885 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:50:33.885 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:50:33.885 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:33.885 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:50:34.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache 2024-08-30T15:50:34.508 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-799.ge5753ca8.el9.n 4/5 2024-08-30T15:50:34.559 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-799.ge5753ca8.el9.noarch 5/5 2024-08-30T15:50:34.662 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:54 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:50:34.798 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:50:34.799 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:34.799 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:50:34.799 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:34.799 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:50:34.799 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 143 k 2024-08-30T15:50:34.799 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:34.799 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:50:34.800 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:34.800 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:50:34.800 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:34.800 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 143 k 2024-08-30T15:50:34.800 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 395 k 2024-08-30T15:50:34.800 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:50:35.008 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-18.2.4-799.ge5753ca 685 kB/s | 143 kB 00:00 2024-08-30T15:50:35.010 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:35.010 INFO:teuthology.orchestra.run.smithi050.stdout:Total 679 kB/s | 143 kB 00:00 2024-08-30T15:50:35.010 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:50:35.018 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:50:35.018 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:50:35.077 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:50:35.078 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:50:35.220 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:50:35.318 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-799.ge5753ca8.e 1/1 2024-08-30T15:50:35.343 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-799.ge5753ca8.e 1/1 2024-08-30T15:50:35.343 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:50:35.343 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-30T15:50:35.343 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-799.ge5753ca8.el9.noarch 5/5 2024-08-30T15:50:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-799.ge5753ca8.el9.n 1/5 2024-08-30T15:50:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-799.ge5753ca8.el9.noarch 2/5 2024-08-30T15:50:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-799.ge5753ca8.el9.no 3/5 2024-08-30T15:50:35.720 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-08-30T15:50:35.968 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-08-30T15:50:35.968 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:35.968 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:50:35.968 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:50:35.968 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:50:35.969 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:50:35.969 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-30T15:50:35.969 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-30T15:50:35.969 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:35.969 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:50:36.175 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-30T15:50:36.280 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-799.ge5753ca8.e 1/1 2024-08-30T15:50:36.280 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:36.280 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:50:36.280 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:50:36.280 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:36.280 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:50:36.528 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr 2024-08-30T15:50:36.719 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:59 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:50:36.850 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:50:36.851 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-08-30T15:50:36.851 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-08-30T15:50:36.851 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-08-30T15:50:36.851 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:50:36.851 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 7.4 M 2024-08-30T15:50:36.852 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-08-30T15:50:36.852 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-30T15:50:36.852 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-30T15:50:36.852 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-30T15:50:36.852 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-08-30T15:50:36.852 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-08-30T15:50:36.852 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-08-30T15:50:36.852 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-08-30T15:50:36.852 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-08-30T15:50:36.852 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-30T15:50:36.852 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-30T15:50:36.852 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-30T15:50:36.853 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:36.853 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:50:36.853 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-08-30T15:50:36.853 INFO:teuthology.orchestra.run.smithi094.stdout:Install 12 Packages 2024-08-30T15:50:36.853 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:36.853 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 38 M 2024-08-30T15:50:36.853 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 212 M 2024-08-30T15:50:36.854 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:50:37.098 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:57 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:50:37.190 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-mgr-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:50:37.233 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:50:37.234 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-08-30T15:50:37.234 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:50:37.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-dashboard 2024-08-30T15:50:37.587 INFO:teuthology.orchestra.run.smithi094.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.4-79 17 MB/s | 7.4 MB 00:00 2024-08-30T15:50:37.654 INFO:teuthology.orchestra.run.smithi094.stdout:(2/12): libquadmath-11.5.0-2.el9.x86_64.rpm 381 kB/s | 191 kB 00:00 2024-08-30T15:50:37.865 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:57 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:50:37.905 INFO:teuthology.orchestra.run.smithi094.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 801 kB 00:00 2024-08-30T15:50:38.001 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:50:38.001 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:38.001 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-08-30T15:50:38.001 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:38.002 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:50:38.002 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 3.5 M 2024-08-30T15:50:38.002 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-08-30T15:50:38.002 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 24 k 2024-08-30T15:50:38.002 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 16 k 2024-08-30T15:50:38.002 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-30T15:50:38.002 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-30T15:50:38.002 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:38.002 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:50:38.002 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:38.003 INFO:teuthology.orchestra.run.smithi050.stdout:Install 5 Packages 2024-08-30T15:50:38.003 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:38.003 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.8 M 2024-08-30T15:50:38.003 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 87 M 2024-08-30T15:50:38.003 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:50:38.169 INFO:teuthology.orchestra.run.smithi094.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 56 kB/s | 33 kB 00:00 2024-08-30T15:50:38.386 INFO:teuthology.orchestra.run.smithi094.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 38 kB/s | 18 kB 00:00 2024-08-30T15:50:38.487 INFO:teuthology.orchestra.run.smithi050.stdout:(1/5): ceph-grafana-dashboards-18.2.4-799.ge575 172 kB/s | 24 kB 00:00 2024-08-30T15:50:38.512 INFO:teuthology.orchestra.run.smithi050.stdout:(2/5): ceph-prometheus-alerts-18.2.4-799.ge5753 99 kB/s | 16 kB 00:00 2024-08-30T15:50:38.579 INFO:teuthology.orchestra.run.smithi050.stdout:(3/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-08-30T15:50:38.604 INFO:teuthology.orchestra.run.smithi050.stdout:(4/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 262 kB/s | 31 kB 00:00 2024-08-30T15:50:38.703 INFO:teuthology.orchestra.run.smithi094.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2024-08-30T15:50:38.741 INFO:teuthology.orchestra.run.smithi050.stdout:(5/5): ceph-mgr-dashboard-18.2.4-799.ge5753ca8. 8.9 MB/s | 3.5 MB 00:00 2024-08-30T15:50:38.745 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:38.745 INFO:teuthology.orchestra.run.smithi050.stdout:Total 5.1 MB/s | 3.8 MB 00:00 2024-08-30T15:50:38.745 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:50:38.753 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:50:38.753 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:50:38.769 INFO:teuthology.orchestra.run.smithi094.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 64 kB/s | 38 kB 00:00 2024-08-30T15:50:38.818 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:50:38.818 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:50:39.072 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:50:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-08-30T15:50:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-08-30T15:50:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-799.ge5753ca8.el9.no 3/5 2024-08-30T15:50:39.520 INFO:teuthology.orchestra.run.smithi094.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.4 MB/s | 4.9 MB 00:01 2024-08-30T15:50:39.545 INFO:teuthology.orchestra.run.smithi094.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 291 kB/s | 245 kB 00:00 2024-08-30T15:50:39.912 INFO:teuthology.orchestra.run.smithi094.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.5 MB/s | 5.1 MB 00:01 2024-08-30T15:50:40.342 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-799.ge5753ca8.el9.n 4/5 2024-08-30T15:50:40.345 INFO:teuthology.orchestra.run.smithi094.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 304 kB/s | 250 kB 00:00 2024-08-30T15:50:40.388 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-799.ge5753ca8.el9.noarch 5/5 2024-08-30T15:50:40.987 INFO:teuthology.orchestra.run.smithi094.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-08-30T15:50:40.989 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:40.989 INFO:teuthology.orchestra.run.smithi094.stdout:Total 9.2 MB/s | 38 MB 00:04 2024-08-30T15:50:41.345 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:50:41.371 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:50:41.371 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:50:41.471 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-799.ge5753ca8.el9.noarch 5/5 2024-08-30T15:50:41.471 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-799.ge5753ca8.el9.n 1/5 2024-08-30T15:50:41.471 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-799.ge5753ca8.el9.noarch 2/5 2024-08-30T15:50:41.471 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-799.ge5753ca8.el9.no 3/5 2024-08-30T15:50:41.471 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-08-30T15:50:41.721 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-08-30T15:50:41.721 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:41.721 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:50:41.721 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:50:41.721 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:50:41.722 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:50:41.722 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-30T15:50:41.722 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-30T15:50:41.722 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:41.722 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:50:41.779 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:50:41.779 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:50:41.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-30T15:50:42.492 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:02 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:50:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:50:42.600 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-30T15:50:42.632 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:50:42.634 INFO:teuthology.orchestra.run.smithi050.stdout:====================================================================================== 2024-08-30T15:50:42.634 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-08-30T15:50:42.634 INFO:teuthology.orchestra.run.smithi050.stdout:====================================================================================== 2024-08-30T15:50:42.634 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:50:42.634 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 7.4 M 2024-08-30T15:50:42.634 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-08-30T15:50:42.634 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-30T15:50:42.635 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-30T15:50:42.635 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-30T15:50:42.635 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-08-30T15:50:42.635 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-08-30T15:50:42.635 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-08-30T15:50:42.635 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-08-30T15:50:42.635 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-08-30T15:50:42.635 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-30T15:50:42.635 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-30T15:50:42.635 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-30T15:50:42.635 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:42.635 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:50:42.635 INFO:teuthology.orchestra.run.smithi050.stdout:====================================================================================== 2024-08-30T15:50:42.636 INFO:teuthology.orchestra.run.smithi050.stdout:Install 12 Packages 2024-08-30T15:50:42.636 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:42.636 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 38 M 2024-08-30T15:50:42.636 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 212 M 2024-08-30T15:50:42.636 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:50:42.637 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-08-30T15:50:42.735 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-08-30T15:50:42.778 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-08-30T15:50:42.961 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-08-30T15:50:42.994 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-08-30T15:50:43.070 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-30T15:50:43.215 INFO:teuthology.orchestra.run.smithi050.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.4-79 16 MB/s | 7.4 MB 00:00 2024-08-30T15:50:43.323 INFO:teuthology.orchestra.run.smithi050.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 302 kB/s | 33 kB 00:00 2024-08-30T15:50:43.349 INFO:teuthology.orchestra.run.smithi050.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 324 kB/s | 191 kB 00:00 2024-08-30T15:50:43.432 INFO:teuthology.orchestra.run.smithi050.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 218 kB/s | 18 kB 00:00 2024-08-30T15:50:43.516 INFO:teuthology.orchestra.run.smithi050.stdout:(5/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 801 kB 00:00 2024-08-30T15:50:43.541 INFO:teuthology.orchestra.run.smithi050.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 355 kB/s | 38 kB 00:00 2024-08-30T15:50:43.713 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-30T15:50:43.741 INFO:teuthology.orchestra.run.smithi050.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-08-30T15:50:43.841 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-30T15:50:44.142 INFO:teuthology.orchestra.run.smithi050.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.7 MB/s | 3.0 MB 00:00 2024-08-30T15:50:44.225 INFO:teuthology.orchestra.run.smithi050.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-08-30T15:50:44.734 INFO:teuthology.orchestra.run.smithi050.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.1 MB/s | 4.9 MB 00:01 2024-08-30T15:50:44.942 INFO:teuthology.orchestra.run.smithi050.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.2 MB/s | 5.1 MB 00:01 2024-08-30T15:50:45.067 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-30T15:50:46.084 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-30T15:50:46.111 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-799.ge5753c 12/12 2024-08-30T15:50:46.701 INFO:teuthology.orchestra.run.smithi050.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.6 MB/s | 16 MB 00:02 2024-08-30T15:50:46.701 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:46.701 INFO:teuthology.orchestra.run.smithi050.stdout:Total 9.4 MB/s | 38 MB 00:04 2024-08-30T15:50:47.069 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:50:47.096 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:50:47.097 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:50:47.527 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:50:47.528 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:50:48.314 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:50:48.408 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-30T15:50:48.449 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-08-30T15:50:48.562 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-08-30T15:50:48.601 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-08-30T15:50:48.657 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-799.ge5753c 12/12 2024-08-30T15:50:48.657 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-799.ge5753c 1/12 2024-08-30T15:50:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-08-30T15:50:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-08-30T15:50:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-30T15:50:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-30T15:50:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-30T15:50:48.658 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-08-30T15:50:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-08-30T15:50:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-08-30T15:50:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-30T15:50:48.659 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-30T15:50:48.793 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-08-30T15:50:48.824 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-08-30T15:50:48.905 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-30T15:50:49.078 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-30T15:50:49.078 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:49.078 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:50:49.078 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:50:49.078 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-30T15:50:49.078 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-30T15:50:49.078 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-30T15:50:49.079 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-08-30T15:50:49.079 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-08-30T15:50:49.079 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.26-2.el9.x86_64 2024-08-30T15:50:49.079 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-08-30T15:50:49.079 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-08-30T15:50:49.079 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-30T15:50:49.079 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-30T15:50:49.079 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-30T15:50:49.079 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:49.079 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:50:49.455 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-rook 2024-08-30T15:50:49.582 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-30T15:50:49.691 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-30T15:50:50.011 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:13 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:50:50.147 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:50:50.148 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:50.148 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-08-30T15:50:50.148 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:50.149 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:50:50.149 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 49 k 2024-08-30T15:50:50.149 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-08-30T15:50:50.149 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-30T15:50:50.149 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-30T15:50:50.149 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-08-30T15:50:50.149 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-30T15:50:50.149 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-30T15:50:50.149 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-30T15:50:50.149 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-30T15:50:50.149 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-30T15:50:50.149 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-30T15:50:50.149 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-30T15:50:50.150 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-30T15:50:50.150 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-30T15:50:50.150 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:50.150 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:50:50.150 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:50.150 INFO:teuthology.orchestra.run.smithi094.stdout:Install 13 Packages 2024-08-30T15:50:50.150 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:50.150 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 2.2 M 2024-08-30T15:50:50.151 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 26 M 2024-08-30T15:50:50.151 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:50:50.488 INFO:teuthology.orchestra.run.smithi094.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 175 kB/s | 19 kB 00:00 2024-08-30T15:50:50.513 INFO:teuthology.orchestra.run.smithi094.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 190 kB/s | 26 kB 00:00 2024-08-30T15:50:50.539 INFO:teuthology.orchestra.run.smithi094.stdout:(3/13): ceph-mgr-rook-18.2.4-799.ge5753ca8.el9. 308 kB/s | 49 kB 00:00 2024-08-30T15:50:50.639 INFO:teuthology.orchestra.run.smithi094.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-08-30T15:50:50.664 INFO:teuthology.orchestra.run.smithi094.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 222 kB 00:00 2024-08-30T15:50:50.690 INFO:teuthology.orchestra.run.smithi094.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-08-30T15:50:50.731 INFO:teuthology.orchestra.run.smithi094.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-08-30T15:50:50.757 INFO:teuthology.orchestra.run.smithi094.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 349 kB/s | 32 kB 00:00 2024-08-30T15:50:50.790 INFO:teuthology.orchestra.run.smithi094.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-08-30T15:50:50.824 INFO:teuthology.orchestra.run.smithi094.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-08-30T15:50:50.858 INFO:teuthology.orchestra.run.smithi094.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-08-30T15:50:50.883 INFO:teuthology.orchestra.run.smithi094.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.5 MB/s | 220 kB 00:00 2024-08-30T15:50:50.916 INFO:teuthology.orchestra.run.smithi094.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.7 MB/s | 1.0 MB 00:00 2024-08-30T15:50:50.919 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:50.919 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.9 MB/s | 2.2 MB 00:00 2024-08-30T15:50:50.982 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:50:50.987 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-30T15:50:50.998 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:50:50.998 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:50:51.087 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:50:51.087 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:50:51.589 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:50:51.760 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-30T15:50:51.809 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-30T15:50:51.865 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-30T15:50:51.902 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-30T15:50:51.943 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-30T15:50:52.044 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-30T15:50:52.072 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-30T15:50:52.098 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-799.ge5753c 12/12 2024-08-30T15:50:52.154 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-08-30T15:50:52.204 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-30T15:50:52.909 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-30T15:50:52.968 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-30T15:50:53.002 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-30T15:50:53.050 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-30T15:50:53.072 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:18.2.4-799.ge5753ca8.el9.noarch 13/13 2024-08-30T15:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-799.ge5753ca8.el9.noarch 13/13 2024-08-30T15:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:18.2.4-799.ge5753ca8.el9.noarch 1/13 2024-08-30T15:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-30T15:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-30T15:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-30T15:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-30T15:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-30T15:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-30T15:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-30T15:50:53.897 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-30T15:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-08-30T15:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-30T15:50:53.898 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-30T15:50:54.209 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-30T15:50:54.209 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:54.209 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:50:54.209 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:50:54.209 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-30T15:50:54.210 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-30T15:50:54.210 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-08-30T15:50:54.210 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-30T15:50:54.210 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-30T15:50:54.210 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-30T15:50:54.210 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-30T15:50:54.210 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-30T15:50:54.210 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-30T15:50:54.210 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-30T15:50:54.210 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-30T15:50:54.210 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-30T15:50:54.210 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:54.210 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:50:54.432 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-cephadm 2024-08-30T15:50:54.641 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-799.ge5753c 12/12 2024-08-30T15:50:54.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-799.ge5753c 1/12 2024-08-30T15:50:54.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-08-30T15:50:54.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-08-30T15:50:54.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-30T15:50:54.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-30T15:50:54.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-30T15:50:54.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-08-30T15:50:54.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-08-30T15:50:54.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-08-30T15:50:54.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-30T15:50:54.642 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-30T15:50:54.985 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:17 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:50:54.999 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-30T15:50:54.999 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:54.999 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:50:54.999 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:50:54.999 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-30T15:50:54.999 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-30T15:50:54.999 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-30T15:50:54.999 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-08-30T15:50:54.999 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-08-30T15:50:55.000 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.26-2.el9.x86_64 2024-08-30T15:50:55.000 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-08-30T15:50:55.000 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-08-30T15:50:55.000 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-30T15:50:55.000 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-30T15:50:55.000 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-30T15:50:55.000 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:55.000 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:50:55.117 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:50:55.117 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:55.118 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-08-30T15:50:55.118 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:55.118 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:50:55.118 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 139 k 2024-08-30T15:50:55.118 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-08-30T15:50:55.118 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-30T15:50:55.118 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-30T15:50:55.118 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-30T15:50:55.118 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-30T15:50:55.118 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-30T15:50:55.118 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:55.118 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:50:55.118 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:50:55.119 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2024-08-30T15:50:55.119 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:55.119 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 7.0 M 2024-08-30T15:50:55.119 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 33 M 2024-08-30T15:50:55.119 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:50:55.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-rook 2024-08-30T15:50:55.412 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): ceph-mgr-cephadm-18.2.4-799.ge5753ca8.el 645 kB/s | 139 kB 00:00 2024-08-30T15:50:55.512 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.4 MB/s | 548 kB 00:00 2024-08-30T15:50:55.546 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2024-08-30T15:50:55.596 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-08-30T15:50:55.779 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 426 kB/s | 248 kB 00:00 2024-08-30T15:50:55.941 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:15 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:50:56.084 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:50:56.085 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:56.086 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-08-30T15:50:56.086 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:56.086 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:50:56.086 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 49 k 2024-08-30T15:50:56.086 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-08-30T15:50:56.086 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-30T15:50:56.086 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-30T15:50:56.086 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-08-30T15:50:56.086 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-30T15:50:56.086 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-30T15:50:56.086 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-30T15:50:56.086 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-30T15:50:56.087 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-30T15:50:56.087 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-30T15:50:56.087 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-30T15:50:56.087 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-30T15:50:56.087 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-30T15:50:56.087 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:56.087 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:50:56.087 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:50:56.087 INFO:teuthology.orchestra.run.smithi050.stdout:Install 13 Packages 2024-08-30T15:50:56.087 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:50:56.088 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.2 M 2024-08-30T15:50:56.088 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 26 M 2024-08-30T15:50:56.088 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:50:56.491 INFO:teuthology.orchestra.run.smithi050.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 242 kB/s | 19 kB 00:00 2024-08-30T15:50:56.524 INFO:teuthology.orchestra.run.smithi050.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 227 kB/s | 26 kB 00:00 2024-08-30T15:50:56.574 INFO:teuthology.orchestra.run.smithi050.stdout:(3/13): ceph-mgr-rook-18.2.4-799.ge5753ca8.el9. 302 kB/s | 49 kB 00:00 2024-08-30T15:50:56.599 INFO:teuthology.orchestra.run.smithi050.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-08-30T15:50:56.625 INFO:teuthology.orchestra.run.smithi050.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-08-30T15:50:56.642 INFO:teuthology.orchestra.run.smithi050.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.2 MB/s | 54 kB 00:00 2024-08-30T15:50:56.675 INFO:teuthology.orchestra.run.smithi050.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 640 kB/s | 32 kB 00:00 2024-08-30T15:50:56.700 INFO:teuthology.orchestra.run.smithi050.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 241 kB/s | 14 kB 00:00 2024-08-30T15:50:56.726 INFO:teuthology.orchestra.run.smithi050.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.8 MB/s | 279 kB 00:00 2024-08-30T15:50:56.768 INFO:teuthology.orchestra.run.smithi050.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.4 MB/s | 220 kB 00:00 2024-08-30T15:50:56.802 INFO:teuthology.orchestra.run.smithi050.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.9 MB/s | 90 kB 00:00 2024-08-30T15:50:56.827 INFO:teuthology.orchestra.run.smithi050.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-08-30T15:50:56.877 INFO:teuthology.orchestra.run.smithi050.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.8 MB/s | 1.0 MB 00:00 2024-08-30T15:50:56.880 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:56.880 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.8 MB/s | 2.2 MB 00:00 2024-08-30T15:50:56.943 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:50:56.959 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:50:56.959 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:50:57.013 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.3 MB/s | 6.0 MB 00:01 2024-08-30T15:50:57.015 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:50:57.015 INFO:teuthology.orchestra.run.smithi094.stdout:Total 3.7 MB/s | 7.0 MB 00:01 2024-08-30T15:50:57.047 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:50:57.048 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:50:57.088 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:50:57.100 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:50:57.100 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:50:57.197 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:50:57.197 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:50:57.494 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:50:57.555 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:50:57.660 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-30T15:50:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-30T15:50:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-30T15:50:57.805 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-30T15:50:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-30T15:50:57.905 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-30T15:50:57.955 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-30T15:50:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-30T15:50:58.187 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-08-30T15:50:58.230 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-30T15:50:58.240 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-30T15:50:58.309 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-30T15:50:58.371 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-30T15:50:58.394 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 6/6 2024-08-30T15:50:58.995 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-30T15:50:59.045 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-30T15:50:59.087 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-30T15:50:59.136 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-30T15:50:59.165 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:18.2.4-799.ge5753ca8.el9.noarch 13/13 2024-08-30T15:50:59.177 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 6/6 2024-08-30T15:50:59.177 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 1/6 2024-08-30T15:50:59.177 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-30T15:50:59.177 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-30T15:50:59.177 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-30T15:50:59.177 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-30T15:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-30T15:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-30T15:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-30T15:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-30T15:50:59.426 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-30T15:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-30T15:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:50:59.427 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:50:59.663 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-fuse 2024-08-30T15:50:59.944 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-799.ge5753ca8.el9.noarch 13/13 2024-08-30T15:50:59.945 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:18.2.4-799.ge5753ca8.el9.noarch 1/13 2024-08-30T15:50:59.945 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-30T15:50:59.945 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-30T15:50:59.945 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-30T15:50:59.945 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-30T15:50:59.945 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-30T15:50:59.945 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-30T15:50:59.945 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-30T15:50:59.945 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-30T15:50:59.945 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-08-30T15:50:59.945 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-30T15:50:59.945 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-30T15:51:00.211 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:23 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:00.269 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-30T15:51:00.269 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:00.269 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:51:00.269 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:51:00.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-30T15:51:00.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-30T15:51:00.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-08-30T15:51:00.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-30T15:51:00.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-30T15:51:00.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-30T15:51:00.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-30T15:51:00.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-30T15:51:00.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-30T15:51:00.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-30T15:51:00.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-30T15:51:00.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-30T15:51:00.271 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:00.271 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-08-30T15:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:51:00.344 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 852 k 2024-08-30T15:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-08-30T15:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-30T15:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-08-30T15:51:00.345 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 932 k 2024-08-30T15:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 2.7 M 2024-08-30T15:51:00.346 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:51:00.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-cephadm 2024-08-30T15:51:00.732 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 617 kB/s | 80 kB 00:00 2024-08-30T15:51:00.874 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-fuse-18.2.4-799.ge5753ca8.el9.x86_6 3.1 MB/s | 852 kB 00:00 2024-08-30T15:51:00.874 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:00.874 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.7 MB/s | 932 kB 00:00 2024-08-30T15:51:00.886 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:51:00.908 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:51:00.908 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:51:00.985 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:51:00.986 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:51:01.086 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:21 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:01.174 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:51:01.223 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:01.224 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:01.224 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-08-30T15:51:01.224 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:01.224 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:51:01.224 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 139 k 2024-08-30T15:51:01.224 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-08-30T15:51:01.225 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-30T15:51:01.225 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-30T15:51:01.225 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-30T15:51:01.225 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-30T15:51:01.225 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-30T15:51:01.225 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:01.225 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:51:01.225 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:01.225 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2024-08-30T15:51:01.225 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:01.226 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 7.0 M 2024-08-30T15:51:01.226 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 33 M 2024-08-30T15:51:01.226 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:51:01.305 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-30T15:51:01.372 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 2/2 2024-08-30T15:51:01.796 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-08-30T15:51:01.830 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): ceph-mgr-cephadm-18.2.4-799.ge5753ca8.el 673 kB/s | 139 kB 00:00 2024-08-30T15:51:01.889 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 991 kB/s | 58 kB 00:00 2024-08-30T15:51:01.914 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-08-30T15:51:01.948 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.5 MB/s | 548 kB 00:00 2024-08-30T15:51:02.031 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-08-30T15:51:02.034 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:02.034 INFO:teuthology.orchestra.run.smithi050.stdout:Total 8.7 MB/s | 7.0 MB 00:00 2024-08-30T15:51:02.039 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 2/2 2024-08-30T15:51:02.039 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 1/2 2024-08-30T15:51:02.107 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:51:02.118 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:51:02.119 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:51:02.220 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:51:02.221 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:51:02.332 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-30T15:51:02.333 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:02.333 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:51:02.333 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-30T15:51:02.333 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:02.333 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:02.532 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:51:02.620 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-volume 2024-08-30T15:51:02.714 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-30T15:51:02.767 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-30T15:51:03.187 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:26 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:03.319 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:03.319 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:03.319 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-08-30T15:51:03.320 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:03.320 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:51:03.320 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 264 k 2024-08-30T15:51:03.320 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-08-30T15:51:03.320 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-30T15:51:03.320 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:03.320 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:51:03.320 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:03.320 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-08-30T15:51:03.320 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:03.320 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 342 k 2024-08-30T15:51:03.320 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 1.5 M 2024-08-30T15:51:03.320 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:51:03.335 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-30T15:51:03.422 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-30T15:51:03.500 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-30T15:51:03.513 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 817 kB/s | 77 kB 00:00 2024-08-30T15:51:03.529 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 6/6 2024-08-30T15:51:03.638 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-volume-18.2.4-799.ge5753ca8.el9.noa 1.2 MB/s | 264 kB 00:00 2024-08-30T15:51:03.638 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:03.638 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.0 MB/s | 342 kB 00:00 2024-08-30T15:51:03.653 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:51:03.661 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:51:03.661 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:51:03.690 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:51:03.690 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:51:03.834 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:51:04.013 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-08-30T15:51:04.060 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-volume-2:18.2.4-799.ge5753ca8.el9.noarch 2/2 2024-08-30T15:51:04.073 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-volume-2:18.2.4-799.ge5753ca8.el9.noarch 2/2 2024-08-30T15:51:04.073 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:51:04.073 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-30T15:51:04.073 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:04.297 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 6/6 2024-08-30T15:51:04.298 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 1/6 2024-08-30T15:51:04.298 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-30T15:51:04.298 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-30T15:51:04.298 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-30T15:51:04.298 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-30T15:51:04.554 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-30T15:51:04.554 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:04.554 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:51:04.554 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:51:04.554 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-30T15:51:04.554 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-30T15:51:04.555 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-30T15:51:04.555 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-30T15:51:04.555 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-30T15:51:04.555 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:04.555 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:04.767 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-volume-2:18.2.4-799.ge5753ca8.el9.noarch 1/2 2024-08-30T15:51:04.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse 2024-08-30T15:51:05.093 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-08-30T15:51:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:05.094 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:51:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:51:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-30T15:51:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:05.094 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:05.345 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:25 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:05.380 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados-devel 2024-08-30T15:51:05.482 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:05.482 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-08-30T15:51:05.482 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:05.482 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:51:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 852 k 2024-08-30T15:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-08-30T15:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-30T15:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2024-08-30T15:51:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:05.484 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 932 k 2024-08-30T15:51:05.484 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.7 M 2024-08-30T15:51:05.484 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:51:05.683 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 530 kB/s | 80 kB 00:00 2024-08-30T15:51:05.942 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:28 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:06.058 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): ceph-fuse-18.2.4-799.ge5753ca8.el9.x86_6 1.6 MB/s | 852 kB 00:00 2024-08-30T15:51:06.059 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:06.059 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.6 MB/s | 932 kB 00:00 2024-08-30T15:51:06.071 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:51:06.077 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:06.078 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:51:06.078 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:06.078 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:51:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 127 k 2024-08-30T15:51:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:06.079 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:51:06.079 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:06.079 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:51:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:06.079 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 127 k 2024-08-30T15:51:06.079 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 456 k 2024-08-30T15:51:06.079 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:51:06.094 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:51:06.094 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:51:06.177 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:51:06.177 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:51:06.277 INFO:teuthology.orchestra.run.smithi094.stdout:librados-devel-18.2.4-799.ge5753ca8.el9.x86_64. 641 kB/s | 127 kB 00:00 2024-08-30T15:51:06.277 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:06.278 INFO:teuthology.orchestra.run.smithi094.stdout:Total 635 kB/s | 127 kB 00:00 2024-08-30T15:51:06.278 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:51:06.283 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:51:06.283 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:51:06.346 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:51:06.347 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:51:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:51:06.495 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-30T15:51:06.498 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:51:06.567 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 2/2 2024-08-30T15:51:06.589 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:06.989 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:07.241 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:07.241 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:07.241 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:51:07.241 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:51:07.241 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:07.241 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:07.245 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 2/2 2024-08-30T15:51:07.245 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 1/2 2024-08-30T15:51:07.478 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs2 2024-08-30T15:51:07.523 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-30T15:51:07.524 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:07.524 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:51:07.524 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-30T15:51:07.524 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:07.524 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:07.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-volume 2024-08-30T15:51:08.044 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:31 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:08.134 INFO:teuthology.orchestra.run.smithi094.stdout:Package libcephfs2-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:51:08.178 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:08.179 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-08-30T15:51:08.179 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:08.245 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs-devel 2024-08-30T15:51:08.362 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:28 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:08.499 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:08.500 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-08-30T15:51:08.500 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:08.500 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:51:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume noarch 2:18.2.4-799.ge5753ca8.el9 ceph-noarch 264 k 2024-08-30T15:51:08.500 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-08-30T15:51:08.500 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-30T15:51:08.501 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:08.501 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:51:08.501 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:08.501 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2024-08-30T15:51:08.501 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:08.501 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 342 k 2024-08-30T15:51:08.501 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.5 M 2024-08-30T15:51:08.502 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:51:08.800 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:31 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:08.931 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:08.932 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:08.932 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:51:08.932 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:08.932 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:51:08.932 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 31 k 2024-08-30T15:51:08.932 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:08.932 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:51:08.932 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:08.932 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:51:08.932 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:08.933 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 31 k 2024-08-30T15:51:08.933 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 139 k 2024-08-30T15:51:08.933 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:51:08.989 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): ceph-volume-18.2.4-799.ge5753ca8.el9.noa 1.1 MB/s | 264 kB 00:00 2024-08-30T15:51:09.081 INFO:teuthology.orchestra.run.smithi094.stdout:libcephfs-devel-18.2.4-799.ge5753ca8.el9.x86_64 210 kB/s | 31 kB 00:00 2024-08-30T15:51:09.081 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:09.082 INFO:teuthology.orchestra.run.smithi094.stdout:Total 208 kB/s | 31 kB 00:00 2024-08-30T15:51:09.082 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:51:09.085 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:51:09.085 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:51:09.089 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:51:09.090 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:51:09.172 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 185 kB/s | 77 kB 00:00 2024-08-30T15:51:09.173 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:09.173 INFO:teuthology.orchestra.run.smithi050.stdout:Total 508 kB/s | 342 kB 00:00 2024-08-30T15:51:09.181 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:51:09.186 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:51:09.197 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:51:09.197 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:51:09.228 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:51:09.229 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:51:09.301 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:51:09.571 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-08-30T15:51:09.608 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-volume-2:18.2.4-799.ge5753ca8.el9.noarch 2/2 2024-08-30T15:51:09.613 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-volume-2:18.2.4-799.ge5753ca8.el9.noarch 2/2 2024-08-30T15:51:09.625 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:51:09.625 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-30T15:51:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:09.831 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:51:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:51:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:09.832 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:10.011 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados2 2024-08-30T15:51:10.316 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-volume-2:18.2.4-799.ge5753ca8.el9.noarch 1/2 2024-08-30T15:51:10.561 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:33 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:10.593 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-08-30T15:51:10.593 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:10.593 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:51:10.593 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume-2:18.2.4-799.ge5753ca8.el9.noarch 2024-08-30T15:51:10.593 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-30T15:51:10.593 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:10.594 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:10.650 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:51:10.693 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:10.694 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-08-30T15:51:10.694 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:10.760 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librbd1 2024-08-30T15:51:10.908 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel 2024-08-30T15:51:11.310 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:34 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:11.399 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:51:11.441 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:11.442 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-08-30T15:51:11.442 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:11.478 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:31 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:11.503 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rados 2024-08-30T15:51:11.616 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:11.616 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:11.616 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:51:11.617 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:11.617 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:51:11.617 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 127 k 2024-08-30T15:51:11.617 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:11.617 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:51:11.617 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:11.617 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:51:11.617 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:11.618 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 127 k 2024-08-30T15:51:11.618 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 456 k 2024-08-30T15:51:11.618 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:51:11.836 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-18.2.4-799.ge5753ca8.el9.x86_64. 581 kB/s | 127 kB 00:00 2024-08-30T15:51:11.837 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:11.837 INFO:teuthology.orchestra.run.smithi050.stdout:Total 577 kB/s | 127 kB 00:00 2024-08-30T15:51:11.837 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:51:11.845 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:51:11.845 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:51:11.908 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:51:11.909 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:51:12.053 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:35 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:12.066 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:51:12.141 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rados-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:51:12.178 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:12.183 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:12.185 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-08-30T15:51:12.185 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:12.253 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rgw 2024-08-30T15:51:12.605 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:12.799 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:35 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:12.887 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rgw-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:51:12.900 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:12.900 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:12.901 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:51:12.901 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:51:12.901 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:12.901 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:12.929 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:12.930 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-08-30T15:51:12.930 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:12.994 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-cephfs 2024-08-30T15:51:13.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2 2024-08-30T15:51:13.547 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:36 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:13.636 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-cephfs-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:51:13.678 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:13.679 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-08-30T15:51:13.680 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:13.692 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:33 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:13.743 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rbd 2024-08-30T15:51:13.784 INFO:teuthology.orchestra.run.smithi050.stdout:Package libcephfs2-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:51:13.829 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:13.830 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-08-30T15:51:13.830 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:13.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs-devel 2024-08-30T15:51:14.291 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:37 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:14.379 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rbd-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:51:14.421 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:14.422 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-08-30T15:51:14.423 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:14.468 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:34 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:14.485 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-fuse 2024-08-30T15:51:14.606 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:14.606 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:14.606 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:51:14.606 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:14.606 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:51:14.606 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 31 k 2024-08-30T15:51:14.606 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:14.607 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:51:14.607 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:14.607 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:51:14.607 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:14.607 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 31 k 2024-08-30T15:51:14.607 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 139 k 2024-08-30T15:51:14.608 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:51:14.756 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs-devel-18.2.4-799.ge5753ca8.el9.x86_64 209 kB/s | 31 kB 00:00 2024-08-30T15:51:14.757 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:14.757 INFO:teuthology.orchestra.run.smithi050.stdout:Total 207 kB/s | 31 kB 00:00 2024-08-30T15:51:14.757 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:51:14.761 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:51:14.761 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:51:14.765 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:51:14.765 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:51:14.852 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:51:14.963 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:15.036 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:38 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:15.167 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:15.167 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:15.167 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-08-30T15:51:15.168 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:15.168 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:51:15.168 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 87 k 2024-08-30T15:51:15.168 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:15.168 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:51:15.168 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:15.168 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:51:15.168 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:15.169 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 87 k 2024-08-30T15:51:15.169 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 227 k 2024-08-30T15:51:15.169 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:51:15.292 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:15.368 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-fuse-18.2.4-799.ge5753ca8.el9.x86_64.rpm 435 kB/s | 87 kB 00:00 2024-08-30T15:51:15.368 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:15.368 INFO:teuthology.orchestra.run.smithi094.stdout:Total 433 kB/s | 87 kB 00:00 2024-08-30T15:51:15.369 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:51:15.377 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:51:15.377 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:51:15.438 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:51:15.438 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:51:15.509 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:15.509 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:15.509 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:51:15.509 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:51:15.509 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:15.509 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:15.589 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:51:15.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2 2024-08-30T15:51:15.704 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:16.038 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:16.257 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:36 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:16.281 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:16.281 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:16.281 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:51:16.281 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:51:16.282 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:16.282 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:16.350 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:51:16.395 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:16.396 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-08-30T15:51:16.396 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:16.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1 2024-08-30T15:51:16.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-mirror 2024-08-30T15:51:17.039 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:37 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:17.128 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:40 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:17.131 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:51:17.177 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:17.178 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-08-30T15:51:17.178 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:17.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados 2024-08-30T15:51:17.262 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 3.0 M 2024-08-30T15:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:51:17.263 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 3.0 M 2024-08-30T15:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 12 M 2024-08-30T15:51:17.264 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:51:17.597 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-mirror-18.2.4-799.ge5753ca8.el9.x86_64.rpm 9.0 MB/s | 3.0 MB 00:00 2024-08-30T15:51:17.598 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:17.598 INFO:teuthology.orchestra.run.smithi094.stdout:Total 8.9 MB/s | 3.0 MB 00:00 2024-08-30T15:51:17.598 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:51:17.607 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:51:17.607 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:51:17.686 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:51:17.687 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:51:17.813 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:37 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:17.905 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rados-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:51:17.950 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-08-30T15:51:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:18.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw 2024-08-30T15:51:18.102 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:51:18.173 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:18.198 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:18.198 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:51:18.199 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-30T15:51:18.199 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-30T15:51:18.199 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-30T15:51:18.199 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:18.600 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:38 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:18.692 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rgw-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:51:18.738 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:18.739 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-08-30T15:51:18.739 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:18.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs 2024-08-30T15:51:19.307 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:19.307 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:19.307 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:51:19.307 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:51:19.307 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:19.307 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:19.372 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:39 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:19.464 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-cephfs-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:51:19.509 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:19.510 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-08-30T15:51:19.510 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:19.525 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-nbd 2024-08-30T15:51:19.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd 2024-08-30T15:51:20.099 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:43 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:20.146 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:40 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:20.237 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rbd-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:51:20.240 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:20.240 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:20.240 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-08-30T15:51:20.241 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:20.241 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:51:20.241 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 172 k 2024-08-30T15:51:20.241 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:20.241 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:51:20.241 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:20.241 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:51:20.241 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:20.242 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 172 k 2024-08-30T15:51:20.242 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 494 k 2024-08-30T15:51:20.242 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:51:20.282 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:20.283 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-08-30T15:51:20.284 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:20.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse 2024-08-30T15:51:20.452 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-nbd-18.2.4-799.ge5753ca8.el9.x86_64.rpm 821 kB/s | 172 kB 00:00 2024-08-30T15:51:20.452 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:20.453 INFO:teuthology.orchestra.run.smithi094.stdout:Total 813 kB/s | 172 kB 00:00 2024-08-30T15:51:20.453 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:51:20.461 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:51:20.461 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:51:20.527 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:51:20.527 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:51:20.681 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:51:20.784 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:20.932 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:40 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:21.072 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:21.073 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:21.073 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-08-30T15:51:21.073 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:21.073 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:51:21.073 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 87 k 2024-08-30T15:51:21.073 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:21.074 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:51:21.074 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:21.074 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:51:21.074 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:21.074 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 87 k 2024-08-30T15:51:21.074 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 227 k 2024-08-30T15:51:21.074 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:51:21.214 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-nbd-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:21.277 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-18.2.4-799.ge5753ca8.el9.x86_64.rpm 426 kB/s | 87 kB 00:00 2024-08-30T15:51:21.277 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:21.278 INFO:teuthology.orchestra.run.smithi050.stdout:Total 424 kB/s | 87 kB 00:00 2024-08-30T15:51:21.278 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:51:21.285 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:51:21.285 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:51:21.353 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:51:21.353 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:51:21.457 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:21.475 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:51:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:51:21.475 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:21.475 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:21.501 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:51:21.609 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:21.703 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-base-debuginfo 2024-08-30T15:51:21.983 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:22.193 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:22.193 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:22.193 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:51:22.193 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:51:22.194 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:22.194 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:22.267 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:45 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:22.401 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:22.402 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:22.402 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:51:22.402 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:22.402 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:51:22.402 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 67 M 2024-08-30T15:51:22.402 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-08-30T15:51:22.402 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 20 M 2024-08-30T15:51:22.403 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:22.403 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:51:22.403 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:22.403 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-08-30T15:51:22.403 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:22.403 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 87 M 2024-08-30T15:51:22.403 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 483 M 2024-08-30T15:51:22.403 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:51:22.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror 2024-08-30T15:51:23.031 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:43 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:23.169 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:23.169 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:23.170 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:51:23.170 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:23.170 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:51:23.170 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 3.0 M 2024-08-30T15:51:23.170 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:23.170 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:51:23.170 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:23.170 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:51:23.170 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:23.171 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.0 M 2024-08-30T15:51:23.171 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 12 M 2024-08-30T15:51:23.171 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:51:23.922 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-18.2.4-799.ge5753ca8.el9.x86_64.rpm 4.0 MB/s | 3.0 MB 00:00 2024-08-30T15:51:23.923 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:23.923 INFO:teuthology.orchestra.run.smithi050.stdout:Total 4.0 MB/s | 3.0 MB 00:00 2024-08-30T15:51:23.923 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:51:23.934 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:51:23.934 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:51:24.019 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:51:24.020 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:51:24.455 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:51:24.514 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:24.539 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T15:51:24.539 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-30T15:51:24.539 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-30T15:51:24.539 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-30T15:51:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:25.533 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:25.533 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:25.533 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:51:25.533 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:51:25.534 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:25.534 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:25.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd 2024-08-30T15:51:25.964 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): ceph-debuginfo-18.2.4-799.ge5753ca8.el9. 5.7 MB/s | 20 MB 00:03 2024-08-30T15:51:26.335 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:46 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:26.479 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:26.481 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:26.481 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-08-30T15:51:26.481 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:26.481 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:51:26.481 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 172 k 2024-08-30T15:51:26.481 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:26.481 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:51:26.481 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:26.482 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:51:26.482 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:26.482 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 172 k 2024-08-30T15:51:26.482 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 494 k 2024-08-30T15:51:26.482 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:51:27.268 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-18.2.4-799.ge5753ca8.el9.x86_64.rpm 219 kB/s | 172 kB 00:00 2024-08-30T15:51:27.268 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:27.269 INFO:teuthology.orchestra.run.smithi050.stdout:Total 218 kB/s | 172 kB 00:00 2024-08-30T15:51:27.269 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:51:27.277 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:51:27.277 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:51:27.343 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:51:27.343 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:51:27.500 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:51:27.608 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:27.995 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:28.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:51:28.242 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:28.242 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:51:28.243 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:51:28.243 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:28.243 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:28.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base-debuginfo 2024-08-30T15:51:28.731 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-base-debuginfo-18.2.4-799.ge5753ca8 11 MB/s | 67 MB 00:06 2024-08-30T15:51:28.732 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:28.733 INFO:teuthology.orchestra.run.smithi094.stdout:Total 14 MB/s | 87 MB 00:06 2024-08-30T15:51:28.733 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:51:28.738 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:51:28.738 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:51:29.007 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:49 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:29.145 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:29.146 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:29.146 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:51:29.146 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:29.146 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:51:29.146 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 67 M 2024-08-30T15:51:29.146 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-08-30T15:51:29.146 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 20 M 2024-08-30T15:51:29.146 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:29.146 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:51:29.146 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:29.147 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2024-08-30T15:51:29.147 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:29.147 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 87 M 2024-08-30T15:51:29.147 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 483 M 2024-08-30T15:51:29.147 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:51:29.290 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:51:29.291 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:51:32.134 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:51:32.871 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): ceph-debuginfo-18.2.4-799.ge5753ca8.el9. 5.5 MB/s | 20 MB 00:03 2024-08-30T15:51:33.755 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): ceph-base-debuginfo-18.2.4-799.ge5753ca8 15 MB/s | 67 MB 00:04 2024-08-30T15:51:33.757 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:33.758 INFO:teuthology.orchestra.run.smithi050.stdout:Total 19 MB/s | 87 MB 00:04 2024-08-30T15:51:33.758 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:51:33.763 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:51:33.763 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:51:34.343 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:51:34.343 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:51:37.344 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:51:39.104 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/2 2024-08-30T15:51:39.181 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 2/2 2024-08-30T15:51:42.613 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 2/2 2024-08-30T15:51:42.614 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/2 2024-08-30T15:51:42.836 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2/2 2024-08-30T15:51:42.836 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:42.836 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:51:42.836 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:51:42.836 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:51:42.836 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:42.836 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:51:43.055 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-common-debuginfo 2024-08-30T15:51:43.604 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:02:06 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:51:43.735 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:51:43.736 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:43.736 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:51:43.736 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:43.736 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:51:43.736 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 402 M 2024-08-30T15:51:43.736 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:43.736 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:51:43.736 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:51:43.736 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:51:43.736 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:51:43.737 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 402 M 2024-08-30T15:51:43.737 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 1.6 G 2024-08-30T15:51:43.737 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:51:44.779 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/2 2024-08-30T15:51:44.856 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 2/2 2024-08-30T15:51:48.156 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 2/2 2024-08-30T15:51:48.156 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/2 2024-08-30T15:51:48.395 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2/2 2024-08-30T15:51:48.395 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:48.395 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:51:48.396 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:51:48.396 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:51:48.396 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:48.396 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:51:48.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-common-debuginfo 2024-08-30T15:51:49.176 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:09 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:51:49.313 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:51:49.314 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:49.314 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:51:49.314 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:49.314 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:51:49.315 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 402 M 2024-08-30T15:51:49.315 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:49.315 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:51:49.315 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:51:49.315 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:51:49.315 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:51:49.315 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 402 M 2024-08-30T15:51:49.316 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.6 G 2024-08-30T15:51:49.316 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:51:54.879 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-common-debuginfo-18.2.4-799.ge5753ca8.el9. 36 MB/s | 402 MB 00:11 2024-08-30T15:51:54.880 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:51:54.880 INFO:teuthology.orchestra.run.smithi094.stdout:Total 36 MB/s | 402 MB 00:11 2024-08-30T15:51:54.881 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:51:54.885 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:51:54.885 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:51:57.264 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:51:57.265 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:52:04.202 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-common-debuginfo-18.2.4-799.ge5753ca8.el9. 27 MB/s | 402 MB 00:14 2024-08-30T15:52:04.202 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:52:04.203 INFO:teuthology.orchestra.run.smithi050.stdout:Total 27 MB/s | 402 MB 00:14 2024-08-30T15:52:04.203 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:52:04.208 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:52:04.208 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:52:06.708 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:52:06.708 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:52:37.601 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:52:37.857 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:52:45.905 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:52:46.128 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:52:46.128 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:52:46.128 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:52:46.129 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:52:46.129 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:52:46.129 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:52:46.365 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-08-30T15:52:46.919 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:09 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:52:47.051 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:52:47.052 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-08-30T15:52:47.052 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:52:47.052 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-08-30T15:52:47.052 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:52:47.052 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 1.4 M 2024-08-30T15:52:47.052 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:52:47.053 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:52:47.053 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-08-30T15:52:47.053 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:52:47.053 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:52:47.053 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 1.4 M 2024-08-30T15:52:47.053 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 6.2 M 2024-08-30T15:52:47.053 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:52:47.940 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-immutable-object-cache-debuginfo-18.2.4-79 1.6 MB/s | 1.4 MB 00:00 2024-08-30T15:52:47.940 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:52:47.941 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.6 MB/s | 1.4 MB 00:00 2024-08-30T15:52:47.941 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:52:47.944 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:52:47.944 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:52:47.986 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:52:47.987 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:52:48.237 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:52:48.335 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.2.4-799.g 1/1 2024-08-30T15:52:48.713 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.2.4-799.g 1/1 2024-08-30T15:52:48.937 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.2.4-799.g 1/1 2024-08-30T15:52:48.937 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:52:48.937 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:52:48.937 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:52:48.937 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:52:48.937 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:52:49.112 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw-debuginfo 2024-08-30T15:52:49.661 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:12 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:52:49.708 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:52:49.792 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:52:49.793 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:52:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:52:49.793 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:52:49.793 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:52:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 163 M 2024-08-30T15:52:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:52:49.793 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:52:49.793 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:52:49.793 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:52:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:52:49.794 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 163 M 2024-08-30T15:52:49.794 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 683 M 2024-08-30T15:52:49.794 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:52:50.308 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:52:54.376 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-radosgw-debuginfo-18.2.4-799.ge5753ca8.el9 36 MB/s | 163 MB 00:04 2024-08-30T15:52:54.377 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:52:54.377 INFO:teuthology.orchestra.run.smithi094.stdout:Total 36 MB/s | 163 MB 00:04 2024-08-30T15:52:54.377 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:52:54.381 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:52:54.381 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:52:54.996 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:52:55.220 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:52:55.220 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:52:55.220 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:52:55.220 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:52:55.220 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:52:55.220 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:52:55.373 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:52:55.374 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:52:55.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-08-30T15:52:56.020 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:16 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:52:56.160 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:52:56.160 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-08-30T15:52:56.160 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:52:56.160 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-08-30T15:52:56.161 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:52:56.161 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 1.4 M 2024-08-30T15:52:56.161 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:52:56.161 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:52:56.161 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-08-30T15:52:56.161 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:52:56.161 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:52:56.161 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 1.4 M 2024-08-30T15:52:56.161 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 6.2 M 2024-08-30T15:52:56.161 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:52:56.453 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-debuginfo-18.2.4-79 4.8 MB/s | 1.4 MB 00:00 2024-08-30T15:52:56.454 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:52:56.454 INFO:teuthology.orchestra.run.smithi050.stdout:Total 4.8 MB/s | 1.4 MB 00:00 2024-08-30T15:52:56.454 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:52:56.458 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:52:56.458 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:52:56.501 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:52:56.501 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:52:56.787 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:52:56.886 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.2.4-799.g 1/1 2024-08-30T15:52:57.256 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.2.4-799.g 1/1 2024-08-30T15:52:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.2.4-799.g 1/1 2024-08-30T15:52:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:52:57.503 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:52:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:52:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:52:57.503 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:52:57.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw-debuginfo 2024-08-30T15:52:58.243 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:18 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:52:58.380 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:52:58.381 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:52:58.381 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:52:58.381 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:52:58.381 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:52:58.381 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 163 M 2024-08-30T15:52:58.382 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:52:58.382 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:52:58.382 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:52:58.382 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:52:58.382 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:52:58.382 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 163 M 2024-08-30T15:52:58.382 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 683 M 2024-08-30T15:52:58.382 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:53:03.031 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-radosgw-debuginfo-18.2.4-799.ge5753ca8.el9 35 MB/s | 163 MB 00:04 2024-08-30T15:53:03.032 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:53:03.033 INFO:teuthology.orchestra.run.smithi050.stdout:Total 35 MB/s | 163 MB 00:04 2024-08-30T15:53:03.033 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:53:03.037 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:53:03.037 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:53:04.074 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:53:04.074 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:53:11.966 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:53:12.062 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x8 1/1 2024-08-30T15:53:16.565 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x8 1/1 2024-08-30T15:53:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x8 1/1 2024-08-30T15:53:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:53:16.784 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:53:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:53:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:53:16.784 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:53:16.975 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-test-debuginfo 2024-08-30T15:53:17.526 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:03:40 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:53:17.658 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:53:17.658 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:53:17.658 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:53:17.658 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:53:17.659 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:53:17.659 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 585 M 2024-08-30T15:53:17.659 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:53:17.659 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:53:17.659 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:53:17.659 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:53:17.659 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:53:17.659 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 585 M 2024-08-30T15:53:17.659 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 2.6 G 2024-08-30T15:53:17.659 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:53:21.741 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:53:21.845 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x8 1/1 2024-08-30T15:53:26.176 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x8 1/1 2024-08-30T15:53:26.393 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x8 1/1 2024-08-30T15:53:26.393 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:53:26.393 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:53:26.393 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:53:26.393 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:53:26.393 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:53:26.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test-debuginfo 2024-08-30T15:53:27.149 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:47 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:53:27.288 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:53:27.289 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:53:27.289 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:53:27.289 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:53:27.289 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:53:27.289 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 585 M 2024-08-30T15:53:27.289 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:53:27.289 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:53:27.289 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:53:27.289 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:53:27.289 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:53:27.289 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 585 M 2024-08-30T15:53:27.289 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.6 G 2024-08-30T15:53:27.290 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:53:41.077 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-test-debuginfo-18.2.4-799.ge5753ca8.el9.x8 25 MB/s | 585 MB 00:23 2024-08-30T15:53:41.077 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:53:41.078 INFO:teuthology.orchestra.run.smithi094.stdout:Total 25 MB/s | 585 MB 00:23 2024-08-30T15:53:41.078 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:53:41.083 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:53:41.083 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:53:44.529 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:53:44.529 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:53:50.639 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-test-debuginfo-18.2.4-799.ge5753ca8.el9.x8 25 MB/s | 585 MB 00:23 2024-08-30T15:53:50.640 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:53:50.641 INFO:teuthology.orchestra.run.smithi050.stdout:Total 25 MB/s | 585 MB 00:23 2024-08-30T15:53:50.641 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:53:50.646 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:53:50.646 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:53:54.273 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:53:54.273 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:54:45.724 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:54:45.887 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:54:54.988 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-test-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:54:55.230 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:54:55.230 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:54:55.230 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:54:55.230 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:54:55.230 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:54:55.230 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:54:55.480 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-base-debuginfo 2024-08-30T15:54:56.032 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:05:19 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:54:56.120 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-base-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:54:56.163 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:54:56.164 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-08-30T15:54:56.164 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:54:56.230 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-debuginfo 2024-08-30T15:54:56.778 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:05:19 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:54:56.910 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:54:56.911 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:54:56.911 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:54:56.911 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:54:56.911 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:54:56.911 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 19 M 2024-08-30T15:54:56.911 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:54:56.911 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:54:56.911 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:54:56.911 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:54:56.911 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:54:56.911 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 19 M 2024-08-30T15:54:56.911 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 101 M 2024-08-30T15:54:56.912 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:54:58.078 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-mgr-debuginfo-18.2.4-799.ge5753ca8.el9.x86 17 MB/s | 19 MB 00:01 2024-08-30T15:54:58.078 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:54:58.079 INFO:teuthology.orchestra.run.smithi094.stdout:Total 17 MB/s | 19 MB 00:01 2024-08-30T15:54:58.079 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:54:58.082 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:54:58.083 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:54:58.231 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:54:58.232 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:54:59.008 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:54:59.110 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:55:00.418 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:55:00.531 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:01.569 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:55:01.778 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:55:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:55:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:55:01.778 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:55:01.958 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mds-debuginfo 2024-08-30T15:55:02.505 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:05:25 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:55:02.636 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:55:02.636 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:55:02.636 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:55:02.637 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:55:02.637 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:55:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 48 M 2024-08-30T15:55:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:55:02.637 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:55:02.637 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:55:02.637 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:55:02.637 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:55:02.637 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 48 M 2024-08-30T15:55:02.637 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 172 M 2024-08-30T15:55:02.637 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:55:04.873 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-mds-debuginfo-18.2.4-799.ge5753ca8.el9.x86 21 MB/s | 48 MB 00:02 2024-08-30T15:55:04.873 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:55:04.874 INFO:teuthology.orchestra.run.smithi094.stdout:Total 21 MB/s | 48 MB 00:02 2024-08-30T15:55:04.874 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:55:04.878 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:55:04.878 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:55:05.190 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:55:05.191 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:55:07.536 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:55:07.755 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:55:07.755 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:07.756 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:55:07.756 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:55:07.756 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:07.756 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:55:08.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base-debuginfo 2024-08-30T15:55:08.593 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:28 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:55:08.684 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 is already installed. 2024-08-30T15:55:08.729 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:55:08.730 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-08-30T15:55:08.730 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:55:08.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-debuginfo 2024-08-30T15:55:09.369 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:29 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:55:09.505 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:55:09.506 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:55:09.507 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:55:09.507 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:55:09.507 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:55:09.507 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 19 M 2024-08-30T15:55:09.507 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:09.507 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:55:09.508 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:55:09.508 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:55:09.508 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:09.508 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 19 M 2024-08-30T15:55:09.508 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 101 M 2024-08-30T15:55:09.509 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:55:09.891 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:55:09.984 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:10.263 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-mgr-debuginfo-18.2.4-799.ge5753ca8.el9.x86 26 MB/s | 19 MB 00:00 2024-08-30T15:55:10.264 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:55:10.265 INFO:teuthology.orchestra.run.smithi050.stdout:Total 25 MB/s | 19 MB 00:00 2024-08-30T15:55:10.265 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:55:10.269 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:55:10.269 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:55:10.425 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:55:10.425 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:55:11.352 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:11.585 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:11.585 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:55:11.585 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:55:11.585 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:55:11.585 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:55:11.585 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:55:11.769 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mon-debuginfo 2024-08-30T15:55:12.319 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:05:35 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:55:12.450 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:55:12.451 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:55:12.451 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:55:12.451 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:55:12.452 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:55:12.452 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 99 M 2024-08-30T15:55:12.452 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:55:12.452 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:55:12.452 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:55:12.452 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:55:12.452 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:55:12.452 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 99 M 2024-08-30T15:55:12.452 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 450 M 2024-08-30T15:55:12.452 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:55:12.746 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:55:12.853 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:13.790 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:14.035 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:14.035 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:14.035 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:55:14.035 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:55:14.035 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:14.035 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:55:14.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mds-debuginfo 2024-08-30T15:55:14.788 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:34 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:55:14.925 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:55:14.926 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:55:14.926 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:55:14.926 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:55:14.926 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:55:14.926 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 48 M 2024-08-30T15:55:14.926 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:14.926 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:55:14.927 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:55:14.927 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:55:14.927 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:14.927 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 48 M 2024-08-30T15:55:14.927 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 172 M 2024-08-30T15:55:14.927 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:55:15.258 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-mon-debuginfo-18.2.4-799.ge5753ca8.el9.x86 35 MB/s | 99 MB 00:02 2024-08-30T15:55:15.259 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:55:15.259 INFO:teuthology.orchestra.run.smithi094.stdout:Total 35 MB/s | 99 MB 00:02 2024-08-30T15:55:15.260 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:55:15.263 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:55:15.264 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:55:15.873 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:55:15.874 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:55:18.325 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-mds-debuginfo-18.2.4-799.ge5753ca8.el9.x86 14 MB/s | 48 MB 00:03 2024-08-30T15:55:18.325 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:55:18.326 INFO:teuthology.orchestra.run.smithi050.stdout:Total 14 MB/s | 48 MB 00:03 2024-08-30T15:55:18.326 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:55:18.330 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:55:18.330 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:55:18.659 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:55:18.659 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:55:23.649 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:55:23.748 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:25.107 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:25.328 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:25.328 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:25.328 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:55:25.328 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:55:25.328 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:25.328 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:55:25.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mon-debuginfo 2024-08-30T15:55:26.076 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:46 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:55:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:55:26.214 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:55:26.214 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:55:26.214 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:55:26.214 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:55:26.215 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:55:26.215 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 99 M 2024-08-30T15:55:26.215 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:26.215 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:55:26.215 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:55:26.215 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:55:26.215 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:26.216 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 99 M 2024-08-30T15:55:26.216 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 450 M 2024-08-30T15:55:26.216 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:55:26.262 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:29.010 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-mon-debuginfo-18.2.4-799.ge5753ca8.el9.x86 35 MB/s | 99 MB 00:02 2024-08-30T15:55:29.011 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:55:29.011 INFO:teuthology.orchestra.run.smithi050.stdout:Total 35 MB/s | 99 MB 00:02 2024-08-30T15:55:29.012 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:55:29.016 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:55:29.016 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:55:29.349 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:55:29.559 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:55:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:55:29.559 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:55:29.559 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:55:29.658 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:55:29.658 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:55:29.753 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-osd-debuginfo 2024-08-30T15:55:30.303 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:05:53 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:55:30.435 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:55:30.435 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:55:30.436 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:55:30.436 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:55:30.436 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:55:30.436 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 287 M 2024-08-30T15:55:30.436 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:55:30.436 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:55:30.436 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:55:30.436 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:55:30.436 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:55:30.436 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 287 M 2024-08-30T15:55:30.436 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 1.2 G 2024-08-30T15:55:30.437 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:55:38.019 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-osd-debuginfo-18.2.4-799.ge5753ca8.el9.x86 38 MB/s | 287 MB 00:07 2024-08-30T15:55:38.019 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:55:38.020 INFO:teuthology.orchestra.run.smithi094.stdout:Total 38 MB/s | 287 MB 00:07 2024-08-30T15:55:38.020 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:55:38.024 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:55:38.024 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:55:39.729 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:55:39.729 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:55:40.586 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:55:40.675 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:43.568 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:55:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:43.799 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:55:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:55:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:43.799 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:55:43.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-osd-debuginfo 2024-08-30T15:55:44.558 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:06:04 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:55:44.695 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:55:44.696 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:55:44.696 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:55:44.696 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:55:44.696 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:55:44.696 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 287 M 2024-08-30T15:55:44.696 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:44.696 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:55:44.697 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:55:44.697 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:55:44.697 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:55:44.697 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 287 M 2024-08-30T15:55:44.697 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.2 G 2024-08-30T15:55:44.697 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:55:52.253 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-osd-debuginfo-18.2.4-799.ge5753ca8.el9.x86 38 MB/s | 287 MB 00:07 2024-08-30T15:55:52.254 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:55:52.255 INFO:teuthology.orchestra.run.smithi050.stdout:Total 38 MB/s | 287 MB 00:07 2024-08-30T15:55:52.255 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:55:52.259 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:55:52.259 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:55:54.057 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:55:54.057 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:56:08.972 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:56:09.070 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:56:16.964 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:56:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:56:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:17.194 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:56:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:17.194 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:17.194 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:56:17.405 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-fuse-debuginfo 2024-08-30T15:56:17.955 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:06:40 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:56:18.086 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:56:18.087 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:18.087 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:56:18.087 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:18.087 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:56:18.087 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 5.9 M 2024-08-30T15:56:18.087 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:18.087 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:56:18.088 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:18.088 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:56:18.088 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:18.088 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 5.9 M 2024-08-30T15:56:18.088 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 28 M 2024-08-30T15:56:18.088 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:56:18.511 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-fuse-debuginfo-18.2.4-799.ge5753ca8.el9.x8 14 MB/s | 5.9 MB 00:00 2024-08-30T15:56:18.512 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:18.513 INFO:teuthology.orchestra.run.smithi094.stdout:Total 14 MB/s | 5.9 MB 00:00 2024-08-30T15:56:18.513 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:56:18.516 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:56:18.516 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:56:18.586 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:56:18.586 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:56:19.301 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:56:19.391 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:56:19.952 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:56:20.177 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:56:20.177 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:20.177 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:56:20.178 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:20.178 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:20.178 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:56:20.359 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados-devel-debuginfo 2024-08-30T15:56:20.914 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:06:43 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:56:21.048 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:56:21.049 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:56:21.049 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:21.049 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:56:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 108 k 2024-08-30T15:56:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:21.049 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:56:21.049 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:21.049 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:56:21.049 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:21.050 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 108 k 2024-08-30T15:56:21.050 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 440 k 2024-08-30T15:56:21.050 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:56:21.273 INFO:teuthology.orchestra.run.smithi094.stdout:librados-devel-debuginfo-18.2.4-799.ge5753ca8.e 485 kB/s | 108 kB 00:00 2024-08-30T15:56:21.274 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:21.274 INFO:teuthology.orchestra.run.smithi094.stdout:Total 482 kB/s | 108 kB 00:00 2024-08-30T15:56:21.274 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:56:21.277 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:56:21.277 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:56:21.312 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:56:21.313 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:56:21.431 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:56:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-debuginfo-2:18.2.4-799.ge5753ca8.el9. 1/1 2024-08-30T15:56:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados-devel-debuginfo-2:18.2.4-799.ge5753ca8.el9. 1/1 2024-08-30T15:56:22.068 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-debuginfo-2:18.2.4-799.ge5753ca8.el9. 1/1 2024-08-30T15:56:22.068 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:22.068 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:56:22.068 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:22.068 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:22.068 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:56:22.243 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs2-debuginfo 2024-08-30T15:56:22.795 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:06:45 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:56:22.930 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:56:22.931 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:56:22.931 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:22.931 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:56:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 5.2 M 2024-08-30T15:56:22.931 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:22.931 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:56:22.932 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:22.932 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:56:22.932 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:22.932 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 5.2 M 2024-08-30T15:56:22.932 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 24 M 2024-08-30T15:56:22.932 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:56:24.226 INFO:teuthology.orchestra.run.smithi094.stdout:libcephfs2-debuginfo-18.2.4-799.ge5753ca8.el9.x 4.0 MB/s | 5.2 MB 00:01 2024-08-30T15:56:24.227 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:24.227 INFO:teuthology.orchestra.run.smithi094.stdout:Total 4.0 MB/s | 5.2 MB 00:01 2024-08-30T15:56:24.228 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:56:24.232 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:56:24.232 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:56:24.303 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:56:24.319 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:56:24.954 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:56:25.039 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_ 1/1 2024-08-30T15:56:25.150 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:56:25.249 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:56:25.560 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_ 1/1 2024-08-30T15:56:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_ 1/1 2024-08-30T15:56:25.784 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:25.785 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:56:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:25.785 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:25.785 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:56:25.963 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados2-debuginfo 2024-08-30T15:56:26.516 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:06:49 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:56:26.649 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:56:26.650 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:26.650 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:56:26.650 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:26.650 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:56:26.650 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 46 M 2024-08-30T15:56:26.650 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:26.650 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:56:26.651 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:26.651 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:56:26.651 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:26.651 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 46 M 2024-08-30T15:56:26.651 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 257 M 2024-08-30T15:56:26.651 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:56:28.546 INFO:teuthology.orchestra.run.smithi094.stdout:librados2-debuginfo-18.2.4-799.ge5753ca8.el9.x8 24 MB/s | 46 MB 00:01 2024-08-30T15:56:28.546 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:28.547 INFO:teuthology.orchestra.run.smithi094.stdout:Total 24 MB/s | 46 MB 00:01 2024-08-30T15:56:28.547 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:56:28.551 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:56:28.551 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:56:28.856 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:56:28.857 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:56:32.742 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:56:32.966 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:56:32.966 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:32.966 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:56:32.966 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:32.967 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:32.967 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:56:33.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse-debuginfo 2024-08-30T15:56:33.745 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:06:53 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:56:33.881 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:56:33.882 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:33.882 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:56:33.882 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:33.883 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:56:33.883 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 5.9 M 2024-08-30T15:56:33.883 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:33.883 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:56:33.883 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:33.883 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:56:33.883 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:33.883 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 5.9 M 2024-08-30T15:56:33.883 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 28 M 2024-08-30T15:56:33.883 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:56:34.003 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:56:34.083 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:56:34.262 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-fuse-debuginfo-18.2.4-799.ge5753ca8.el9.x8 15 MB/s | 5.9 MB 00:00 2024-08-30T15:56:34.262 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:34.263 INFO:teuthology.orchestra.run.smithi050.stdout:Total 15 MB/s | 5.9 MB 00:00 2024-08-30T15:56:34.263 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:56:34.266 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:56:34.266 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:56:34.337 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:56:34.338 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:56:35.101 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:56:35.210 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:56:35.746 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:56:35.975 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:56:35.975 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:35.975 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:56:35.976 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:35.976 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:35.976 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:56:35.982 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:56:36.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel-debuginfo 2024-08-30T15:56:36.200 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:56:36.200 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:36.200 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:56:36.200 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:36.200 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:36.200 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:56:36.391 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librbd1-debuginfo 2024-08-30T15:56:36.716 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:06:56 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:56:36.852 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:56:36.852 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:36.852 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:56:36.852 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:36.853 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:56:36.853 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 108 k 2024-08-30T15:56:36.853 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:36.853 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:56:36.853 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:36.853 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:56:36.853 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:36.853 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 108 k 2024-08-30T15:56:36.854 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 440 k 2024-08-30T15:56:36.854 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:56:36.942 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:06:59 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:56:37.065 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-debuginfo-18.2.4-799.ge5753ca8.e 514 kB/s | 108 kB 00:00 2024-08-30T15:56:37.065 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:37.065 INFO:teuthology.orchestra.run.smithi050.stdout:Total 510 kB/s | 108 kB 00:00 2024-08-30T15:56:37.066 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:56:37.069 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:56:37.069 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:56:37.077 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:56:37.081 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:56:37.081 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:37.081 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:56:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 51 M 2024-08-30T15:56:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:37.081 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:56:37.081 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:37.081 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:56:37.082 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:37.082 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 51 M 2024-08-30T15:56:37.082 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 292 M 2024-08-30T15:56:37.082 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:56:37.104 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:56:37.105 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:56:37.226 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:56:37.322 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-debuginfo-2:18.2.4-799.ge5753ca8.el9. 1/1 2024-08-30T15:56:37.770 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-debuginfo-2:18.2.4-799.ge5753ca8.el9. 1/1 2024-08-30T15:56:37.982 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-debuginfo-2:18.2.4-799.ge5753ca8.el9. 1/1 2024-08-30T15:56:37.982 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:37.982 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:56:37.982 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:37.982 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:37.982 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:56:38.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2-debuginfo 2024-08-30T15:56:38.632 INFO:teuthology.orchestra.run.smithi094.stdout:librbd1-debuginfo-18.2.4-799.ge5753ca8.el9.x86_ 33 MB/s | 51 MB 00:01 2024-08-30T15:56:38.633 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:38.634 INFO:teuthology.orchestra.run.smithi094.stdout:Total 33 MB/s | 51 MB 00:01 2024-08-30T15:56:38.634 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:56:38.638 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:56:38.638 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:56:38.737 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:06:58 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:56:38.875 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:56:38.875 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:38.875 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:56:38.876 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:38.876 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:56:38.876 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 5.2 M 2024-08-30T15:56:38.876 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:38.876 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:56:38.876 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:38.876 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:56:38.876 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:38.876 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 5.2 M 2024-08-30T15:56:38.876 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 24 M 2024-08-30T15:56:38.876 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:56:38.973 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:56:38.973 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:56:39.259 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs2-debuginfo-18.2.4-799.ge5753ca8.el9.x 14 MB/s | 5.2 MB 00:00 2024-08-30T15:56:39.260 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:39.260 INFO:teuthology.orchestra.run.smithi050.stdout:Total 13 MB/s | 5.2 MB 00:00 2024-08-30T15:56:39.260 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:56:39.264 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:56:39.264 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:56:39.332 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:56:39.332 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:56:40.011 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:56:40.116 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_ 1/1 2024-08-30T15:56:40.617 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_ 1/1 2024-08-30T15:56:40.848 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_ 1/1 2024-08-30T15:56:40.848 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:40.849 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:56:40.849 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:40.849 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:40.849 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:56:41.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2-debuginfo 2024-08-30T15:56:41.616 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:07:01 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:56:41.754 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:56:41.755 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:41.755 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:56:41.755 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:41.755 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:56:41.755 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 46 M 2024-08-30T15:56:41.755 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:41.755 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:56:41.756 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:41.756 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:56:41.756 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:41.756 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 46 M 2024-08-30T15:56:41.756 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 257 M 2024-08-30T15:56:41.756 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:56:43.187 INFO:teuthology.orchestra.run.smithi050.stdout:librados2-debuginfo-18.2.4-799.ge5753ca8.el9.x8 32 MB/s | 46 MB 00:01 2024-08-30T15:56:43.188 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:43.188 INFO:teuthology.orchestra.run.smithi050.stdout:Total 32 MB/s | 46 MB 00:01 2024-08-30T15:56:43.189 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:56:43.193 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:56:43.193 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:56:43.513 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:56:43.513 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:56:44.767 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:56:44.861 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librbd1-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:56:47.027 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:56:47.249 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:56:47.249 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:47.249 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:56:47.250 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:47.250 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:47.250 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:56:47.465 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-cephfs-debuginfo 2024-08-30T15:56:48.019 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:07:11 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:56:48.150 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:56:48.151 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:48.151 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:56:48.151 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:48.151 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:56:48.152 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 615 k 2024-08-30T15:56:48.152 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:48.152 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:56:48.152 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:48.152 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:56:48.152 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:48.152 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 615 k 2024-08-30T15:56:48.152 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 2.1 M 2024-08-30T15:56:48.152 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:56:48.413 INFO:teuthology.orchestra.run.smithi094.stdout:python3-cephfs-debuginfo-18.2.4-799.ge5753ca8.e 2.3 MB/s | 615 kB 00:00 2024-08-30T15:56:48.413 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:48.414 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.3 MB/s | 615 kB 00:00 2024-08-30T15:56:48.414 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:56:48.417 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:56:48.417 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:56:48.455 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:56:48.456 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:56:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:56:48.708 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-debuginfo-2:18.2.4-799.ge5753ca8.el9. 1/1 2024-08-30T15:56:48.981 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:56:49.070 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.2.4-799.ge5753ca8.el9. 1/1 2024-08-30T15:56:49.078 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:56:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-debuginfo-2:18.2.4-799.ge5753ca8.el9. 1/1 2024-08-30T15:56:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:49.300 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:56:49.300 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:49.301 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:49.301 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:56:49.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rados-debuginfo 2024-08-30T15:56:50.051 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:07:13 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:56:50.183 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:56:50.184 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:50.184 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:56:50.184 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:50.184 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:56:50.184 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 769 k 2024-08-30T15:56:50.184 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:50.184 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:56:50.185 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:50.185 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:56:50.185 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:50.185 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 769 k 2024-08-30T15:56:50.185 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 2.7 M 2024-08-30T15:56:50.185 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:56:50.449 INFO:teuthology.orchestra.run.smithi094.stdout:python3-rados-debuginfo-18.2.4-799.ge5753ca8.el 2.8 MB/s | 769 kB 00:00 2024-08-30T15:56:50.449 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:50.450 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.8 MB/s | 769 kB 00:00 2024-08-30T15:56:50.450 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:56:50.453 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:56:50.453 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:56:50.493 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:56:50.493 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:56:50.669 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:56:50.779 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-debuginfo-2:18.2.4-799.ge5753ca8.el9.x 1/1 2024-08-30T15:56:50.954 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:56:51.124 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: python3-rados-debuginfo-2:18.2.4-799.ge5753ca8.el9.x 1/1 2024-08-30T15:56:51.181 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_6 1/1 2024-08-30T15:56:51.181 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:51.182 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:56:51.182 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:51.182 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:51.182 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:56:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-debuginfo-2:18.2.4-799.ge5753ca8.el9.x 1/1 2024-08-30T15:56:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:51.341 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:56:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:51.342 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:51.342 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:56:51.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1-debuginfo 2024-08-30T15:56:51.555 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rbd-debuginfo 2024-08-30T15:56:51.940 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:07:11 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:56:52.080 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:56:52.080 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:52.080 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:56:52.081 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:52.081 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:56:52.081 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 51 M 2024-08-30T15:56:52.081 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:52.081 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:56:52.081 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:56:52.081 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:56:52.081 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:56:52.082 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 51 M 2024-08-30T15:56:52.082 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 292 M 2024-08-30T15:56:52.082 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:56:52.105 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:07:15 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:56:52.236 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:56:52.237 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:52.237 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:56:52.237 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:52.237 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:56:52.237 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 1.1 M 2024-08-30T15:56:52.237 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:52.237 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:56:52.237 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:52.237 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:56:52.237 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:52.237 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 1.1 M 2024-08-30T15:56:52.238 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 4.3 M 2024-08-30T15:56:52.238 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:56:52.952 INFO:teuthology.orchestra.run.smithi094.stdout:python3-rbd-debuginfo-18.2.4-799.ge5753ca8.el9. 1.6 MB/s | 1.1 MB 00:00 2024-08-30T15:56:52.953 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:52.953 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.6 MB/s | 1.1 MB 00:00 2024-08-30T15:56:52.953 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:56:52.956 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:56:52.957 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:56:52.998 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:56:52.999 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:56:53.215 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:56:53.322 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:56:53.632 INFO:teuthology.orchestra.run.smithi050.stdout:librbd1-debuginfo-18.2.4-799.ge5753ca8.el9.x86_ 33 MB/s | 51 MB 00:01 2024-08-30T15:56:53.633 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:53.633 INFO:teuthology.orchestra.run.smithi050.stdout:Total 33 MB/s | 51 MB 00:01 2024-08-30T15:56:53.633 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:56:53.637 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:56:53.638 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:56:53.660 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:56:53.873 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:56:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:53.874 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:56:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:53.874 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:53.874 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:56:53.990 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:56:53.990 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:56:54.069 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rgw-debuginfo 2024-08-30T15:56:54.616 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:07:17 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:56:54.748 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:56:54.748 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:54.749 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:56:54.749 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:54.749 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:56:54.749 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 337 k 2024-08-30T15:56:54.749 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:54.749 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:56:54.749 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:54.749 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:56:54.749 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:54.750 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 337 k 2024-08-30T15:56:54.750 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 1.0 M 2024-08-30T15:56:54.750 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:56:54.980 INFO:teuthology.orchestra.run.smithi094.stdout:python3-rgw-debuginfo-18.2.4-799.ge5753ca8.el9. 1.4 MB/s | 337 kB 00:00 2024-08-30T15:56:54.984 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:54.985 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.4 MB/s | 337 kB 00:00 2024-08-30T15:56:54.985 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:56:54.985 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:56:54.985 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:56:55.021 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:56:55.021 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:56:55.159 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:56:55.243 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:56:55.570 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:56:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:56:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:55.800 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:56:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:55.800 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:56:56.009 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-fuse-debuginfo 2024-08-30T15:56:56.561 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:07:19 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:56:56.693 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:56:56.693 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:56:56.694 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:56.694 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:56:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 512 k 2024-08-30T15:56:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:56.694 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:56:56.694 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:56.694 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:56:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:56.694 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 512 k 2024-08-30T15:56:56.694 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 2.5 M 2024-08-30T15:56:56.694 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:56:56.958 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-fuse-debuginfo-18.2.4-799.ge5753ca8.el9.x86 1.9 MB/s | 512 kB 00:00 2024-08-30T15:56:56.958 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:56:56.958 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.9 MB/s | 512 kB 00:00 2024-08-30T15:56:56.958 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:56:56.962 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:56:56.962 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:56:56.999 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:56:56.999 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:56:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:56:57.303 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:56:57.649 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:56:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:56:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:57.873 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:56:57.873 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:56:57.874 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:57.874 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:56:58.071 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-mirror-debuginfo 2024-08-30T15:56:58.621 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:07:21 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:56:58.750 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:56:58.750 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:58.750 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:56:58.750 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:58.750 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:56:58.750 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 51 M 2024-08-30T15:56:58.751 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:58.751 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:56:58.751 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:56:58.751 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:56:58.751 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:56:58.751 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 51 M 2024-08-30T15:56:58.751 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 282 M 2024-08-30T15:56:58.752 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:57:00.127 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:57:00.222 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librbd1-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:57:00.294 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-mirror-debuginfo-18.2.4-799.ge5753ca8.el9.x 33 MB/s | 51 MB 00:01 2024-08-30T15:57:00.295 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:57:00.295 INFO:teuthology.orchestra.run.smithi094.stdout:Total 33 MB/s | 51 MB 00:01 2024-08-30T15:57:00.296 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:57:00.300 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:57:00.300 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:57:00.633 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:57:00.633 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:57:02.313 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:57:02.539 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:57:02.540 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:02.540 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:57:02.540 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:57:02.540 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:02.540 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:57:02.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs-debuginfo 2024-08-30T15:57:03.316 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:07:23 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:57:03.455 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:57:03.456 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:57:03.456 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:03.456 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:57:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 615 k 2024-08-30T15:57:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:03.457 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:57:03.457 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:03.457 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:57:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:03.457 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 615 k 2024-08-30T15:57:03.457 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.1 M 2024-08-30T15:57:03.457 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:57:03.710 INFO:teuthology.orchestra.run.smithi050.stdout:python3-cephfs-debuginfo-18.2.4-799.ge5753ca8.e 2.4 MB/s | 615 kB 00:00 2024-08-30T15:57:03.710 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:57:03.710 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.4 MB/s | 615 kB 00:00 2024-08-30T15:57:03.710 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:57:03.714 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:57:03.714 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:57:03.753 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:57:03.754 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:57:03.926 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:57:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-debuginfo-2:18.2.4-799.ge5753ca8.el9. 1/1 2024-08-30T15:57:04.364 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.2.4-799.ge5753ca8.el9. 1/1 2024-08-30T15:57:04.610 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-debuginfo-2:18.2.4-799.ge5753ca8.el9. 1/1 2024-08-30T15:57:04.610 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:04.610 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:57:04.610 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:57:04.610 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:04.610 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:57:04.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados-debuginfo 2024-08-30T15:57:05.379 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:07:25 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:57:05.517 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:57:05.518 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:57:05.519 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:05.519 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:57:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 769 k 2024-08-30T15:57:05.519 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:05.519 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:57:05.520 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:05.520 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:57:05.520 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:05.520 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 769 k 2024-08-30T15:57:05.520 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.7 M 2024-08-30T15:57:05.521 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:57:05.782 INFO:teuthology.orchestra.run.smithi050.stdout:python3-rados-debuginfo-18.2.4-799.ge5753ca8.el 2.9 MB/s | 769 kB 00:00 2024-08-30T15:57:05.782 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:57:05.783 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.8 MB/s | 769 kB 00:00 2024-08-30T15:57:05.783 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:57:05.787 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:57:05.787 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:57:05.827 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:57:05.827 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:57:06.009 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:57:06.106 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-debuginfo-2:18.2.4-799.ge5753ca8.el9.x 1/1 2024-08-30T15:57:06.329 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:57:06.431 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_ 1/1 2024-08-30T15:57:06.477 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: python3-rados-debuginfo-2:18.2.4-799.ge5753ca8.el9.x 1/1 2024-08-30T15:57:06.705 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-debuginfo-2:18.2.4-799.ge5753ca8.el9.x 1/1 2024-08-30T15:57:06.705 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:06.705 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:57:06.705 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:57:06.705 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:06.705 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:57:06.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd-debuginfo 2024-08-30T15:57:07.461 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:07:27 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:57:07.600 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:57:07.600 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:07.600 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:57:07.601 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:07.601 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:57:07.601 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 1.1 M 2024-08-30T15:57:07.601 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:07.601 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:57:07.601 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:07.601 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:57:07.601 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:07.601 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 1.1 M 2024-08-30T15:57:07.601 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 4.3 M 2024-08-30T15:57:07.601 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:57:07.893 INFO:teuthology.orchestra.run.smithi050.stdout:python3-rbd-debuginfo-18.2.4-799.ge5753ca8.el9. 3.9 MB/s | 1.1 MB 00:00 2024-08-30T15:57:07.894 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:57:07.894 INFO:teuthology.orchestra.run.smithi050.stdout:Total 3.9 MB/s | 1.1 MB 00:00 2024-08-30T15:57:07.894 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:57:07.898 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:57:07.898 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:57:07.941 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:57:07.941 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:57:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:57:08.276 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:57:08.481 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_ 1/1 2024-08-30T15:57:08.698 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:57:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_ 1/1 2024-08-30T15:57:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:08.702 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:57:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:57:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:08.702 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:57:08.904 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-nbd-debuginfo 2024-08-30T15:57:08.928 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:57:08.929 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:08.929 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:57:08.929 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:57:08.929 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:08.929 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:57:09.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw-debuginfo 2024-08-30T15:57:09.452 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:07:32 ago on Fri 30 Aug 2024 03:49:37 PM UTC. 2024-08-30T15:57:09.583 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-08-30T15:57:09.583 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:57:09.584 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-08-30T15:57:09.584 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:57:09.584 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-08-30T15:57:09.584 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 1.3 M 2024-08-30T15:57:09.584 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:09.584 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-08-30T15:57:09.584 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-08-30T15:57:09.584 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-08-30T15:57:09.584 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:09.584 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 1.3 M 2024-08-30T15:57:09.584 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 5.6 M 2024-08-30T15:57:09.584 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-08-30T15:57:09.688 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:07:29 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:57:09.826 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:57:09.827 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:09.827 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:57:09.827 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:09.827 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:57:09.828 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 337 k 2024-08-30T15:57:09.828 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:09.828 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:57:09.828 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:09.828 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:57:09.828 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:09.828 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 337 k 2024-08-30T15:57:09.828 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.0 M 2024-08-30T15:57:09.828 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:57:09.880 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-nbd-debuginfo-18.2.4-799.ge5753ca8.el9.x86_ 4.3 MB/s | 1.3 MB 00:00 2024-08-30T15:57:09.880 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:57:09.881 INFO:teuthology.orchestra.run.smithi094.stdout:Total 4.3 MB/s | 1.3 MB 00:00 2024-08-30T15:57:09.881 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-08-30T15:57:09.884 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-08-30T15:57:09.884 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-08-30T15:57:09.926 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-08-30T15:57:09.926 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-08-30T15:57:10.064 INFO:teuthology.orchestra.run.smithi050.stdout:python3-rgw-debuginfo-18.2.4-799.ge5753ca8.el9. 1.4 MB/s | 337 kB 00:00 2024-08-30T15:57:10.065 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:57:10.065 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.4 MB/s | 337 kB 00:00 2024-08-30T15:57:10.065 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:57:10.068 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:57:10.068 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:57:10.106 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:57:10.106 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:57:10.159 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-08-30T15:57:10.248 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:57:10.259 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:57:10.346 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:57:10.636 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:57:10.716 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:57:10.845 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:57:10.845 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:10.845 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-08-30T15:57:10.845 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:57:10.845 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:10.845 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-08-30T15:57:10.961 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86 1/1 2024-08-30T15:57:10.961 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:10.961 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:57:10.961 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:57:10.962 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:10.962 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:57:11.053 DEBUG:teuthology.parallel:result is None 2024-08-30T15:57:11.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse-debuginfo 2024-08-30T15:57:11.709 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:07:31 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:57:11.847 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:57:11.847 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:57:11.848 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:11.848 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:57:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 512 k 2024-08-30T15:57:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:11.848 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:57:11.848 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:11.848 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:57:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:11.848 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 512 k 2024-08-30T15:57:11.848 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.5 M 2024-08-30T15:57:11.848 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:57:12.126 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-debuginfo-18.2.4-799.ge5753ca8.el9.x86 1.8 MB/s | 512 kB 00:00 2024-08-30T15:57:12.126 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:57:12.126 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.8 MB/s | 512 kB 00:00 2024-08-30T15:57:12.126 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:57:12.130 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:57:12.130 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:57:12.169 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:57:12.169 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:57:12.382 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:57:12.483 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:57:12.840 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:57:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:57:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:13.081 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:57:13.081 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:57:13.082 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:13.082 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:57:13.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror-debuginfo 2024-08-30T15:57:13.832 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:07:33 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:57:13.970 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:57:13.971 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:13.971 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:57:13.971 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:13.971 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:57:13.971 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 51 M 2024-08-30T15:57:13.971 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:13.971 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:57:13.971 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:13.971 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:57:13.971 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:13.972 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 51 M 2024-08-30T15:57:13.972 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 282 M 2024-08-30T15:57:13.972 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:57:15.521 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-debuginfo-18.2.4-799.ge5753ca8.el9.x 33 MB/s | 51 MB 00:01 2024-08-30T15:57:15.522 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:57:15.522 INFO:teuthology.orchestra.run.smithi050.stdout:Total 33 MB/s | 51 MB 00:01 2024-08-30T15:57:15.522 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:57:15.527 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:57:15.527 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:57:15.873 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:57:15.873 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:57:21.931 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:57:22.035 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_ 1/1 2024-08-30T15:57:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_ 1/1 2024-08-30T15:57:24.252 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_ 1/1 2024-08-30T15:57:24.252 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:24.253 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:57:24.253 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:57:24.253 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:24.253 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:57:24.441 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd-debuginfo 2024-08-30T15:57:25.008 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:07:45 ago on Fri 30 Aug 2024 03:49:40 PM UTC. 2024-08-30T15:57:25.144 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-08-30T15:57:25.145 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:25.145 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-08-30T15:57:25.146 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:25.146 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-08-30T15:57:25.146 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-debuginfo x86_64 2:18.2.4-799.ge5753ca8.el9 ceph 1.3 M 2024-08-30T15:57:25.146 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:25.147 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-08-30T15:57:25.147 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-08-30T15:57:25.147 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-08-30T15:57:25.147 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:25.147 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 1.3 M 2024-08-30T15:57:25.147 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 5.6 M 2024-08-30T15:57:25.148 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-08-30T15:57:25.432 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-debuginfo-18.2.4-799.ge5753ca8.el9.x86_ 4.4 MB/s | 1.3 MB 00:00 2024-08-30T15:57:25.433 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-08-30T15:57:25.433 INFO:teuthology.orchestra.run.smithi050.stdout:Total 4.4 MB/s | 1.3 MB 00:00 2024-08-30T15:57:25.433 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-08-30T15:57:25.437 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-08-30T15:57:25.437 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-08-30T15:57:25.480 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-08-30T15:57:25.480 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-08-30T15:57:25.729 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-08-30T15:57:25.838 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:57:26.212 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:57:26.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 1/1 2024-08-30T15:57:26.447 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:26.447 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-08-30T15:57:26.448 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-debuginfo-2:18.2.4-799.ge5753ca8.el9.x86_64 2024-08-30T15:57:26.448 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:26.448 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-08-30T15:57:26.622 DEBUG:teuthology.parallel:result is None 2024-08-30T15:57:26.623 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e5753ca846ae583f991bb12d29862578d4f67590 2024-08-30T15:57:26.782 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-30T15:57:26.803 INFO:teuthology.orchestra.run.smithi050.stdout:18.2.4-799.ge5753ca8.el9 2024-08-30T15:57:26.804 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-799.ge5753ca8.el9 2024-08-30T15:57:26.804 INFO:teuthology.task.install:The correct ceph version 18.2.4-799.ge5753ca8 is installed. 2024-08-30T15:57:26.806 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e5753ca846ae583f991bb12d29862578d4f67590 2024-08-30T15:57:26.956 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-30T15:57:26.981 INFO:teuthology.orchestra.run.smithi094.stdout:18.2.4-799.ge5753ca8.el9 2024-08-30T15:57:26.982 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-799.ge5753ca8.el9 2024-08-30T15:57:26.982 INFO:teuthology.task.install:The correct ceph version 18.2.4-799.ge5753ca8 is installed. 2024-08-30T15:57:26.983 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-30T15:57:26.984 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:26.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-30T15:57:27.010 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:57:27.010 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-30T15:57:27.051 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-30T15:57:27.051 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:27.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2024-08-30T15:57:27.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-30T15:57:27.140 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:57:27.140 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2024-08-30T15:57:27.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-30T15:57:27.239 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-30T15:57:27.239 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:27.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-30T15:57:27.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-30T15:57:27.330 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:57:27.330 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-30T15:57:27.359 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-30T15:57:27.428 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-30T15:57:27.428 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:27.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2024-08-30T15:57:27.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-30T15:57:27.517 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:57:27.517 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/stdin-killer 2024-08-30T15:57:27.546 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-30T15:57:27.615 INFO:teuthology.run_tasks:Running task ceph... 2024-08-30T15:57:27.692 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-08-30T15:57:27.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /var/log/ceph 2024-08-30T15:57:27.694 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /var/log/ceph 2024-08-30T15:57:27.722 INFO:tasks.ceph:Disabling ceph logrotate... 2024-08-30T15:57:27.722 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-30T15:57:27.759 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-30T15:57:27.789 INFO:tasks.ceph:Creating extra log directories... 2024-08-30T15:57:27.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-30T15:57:27.823 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-30T15:57:27.865 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-08-30T15:57:27.866 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80, 'osd op thread timeout': 150}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW OPS\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-08-30T15:57:27.866 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi/7881803', 'branch': 'wip-ksirivad-backport-reef-57906', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7881803', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-ksirivad-backport-reef-57906'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80, 'osd op thread timeout': 150}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW OPS\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': 'e5753ca846ae583f991bb12d29862578d4f67590', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'e5753ca846ae583f991bb12d29862578d4f67590'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-ksirivad-backport-reef-57906', 'sha1': 'e5753ca846ae583f991bb12d29862578d4f67590'}}, 'owner': 'scheduled_ksirivad@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 819, 'sha1': 'e5753ca846ae583f991bb12d29862578d4f67590', 'sleep_before_teardown': 0, 'subset': '90000/120000', 'suite': 'rados', 'suite_branch': 'wip-ksirivad-backport-reef-57906', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_e5753ca846ae583f991bb12d29862578d4f67590/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'e5753ca846ae583f991bb12d29862578d4f67590', 'targets': {'smithi050.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFI5hI4j98WMgLeNrcLdzDCoQr7AzKV7z6rlHx2+q6qRSRxBzKBR03RGmTdppxhI4ZuBK6bjlA2e+GFjedqrb1s=', 'smithi094.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJM6mdZQusRlJDgXQK2y23aASpUg/lnKj8AM+uOaNq0kEIkRNsTgsUnnxnbVULa7yTGKzo/F3pUBpaGZo8TfjDA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'mon_recovery': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-08-30_14:53:30', 'tube': 'smithi', 'user': 'ksirivad', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-30T15:57:27.866 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-30T15:57:27.888 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-30T15:57:27.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2024-08-30T15:57:27.943 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/run/ceph 2024-08-30T15:57:27.977 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:27.978 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2024-08-30T15:57:28.020 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-30T15:57:28.020 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2024-08-30T15:57:28.077 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-30T15:57:28.077 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T15:57:28.077 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-08-30T15:57:28.077 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T15:57:28.077 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T15:57:28.077 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-08-30 15:51:24.811321493 +0000 2024-08-30T15:57:28.077 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-08-30 15:48:58.431004112 +0000 2024-08-30T15:57:28.078 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-08-30 15:48:58.431004112 +0000 2024-08-30T15:57:28.078 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-08-30 15:48:58.431004112 +0000 2024-08-30T15:57:28.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-30T15:57:28.139 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-08-30T15:57:28.140 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-08-30T15:57:28.140 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.3146e-05 s, 5.5 MB/s 2024-08-30T15:57:28.141 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-30T15:57:28.198 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2024-08-30T15:57:28.253 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-30T15:57:28.253 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T15:57:28.253 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-08-30T15:57:28.253 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T15:57:28.253 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T15:57:28.253 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-08-30 15:51:24.811321493 +0000 2024-08-30T15:57:28.253 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-08-30 15:48:58.744994449 +0000 2024-08-30T15:57:28.253 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-08-30 15:48:58.744994449 +0000 2024-08-30T15:57:28.253 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-08-30 15:48:58.744994449 +0000 2024-08-30T15:57:28.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-30T15:57:28.316 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-08-30T15:57:28.316 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-08-30T15:57:28.316 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.2149e-05 s, 5.6 MB/s 2024-08-30T15:57:28.317 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-30T15:57:28.373 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2024-08-30T15:57:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-30T15:57:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T15:57:28.429 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-08-30T15:57:28.429 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T15:57:28.429 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T15:57:28.429 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-08-30 15:51:24.812321468 +0000 2024-08-30T15:57:28.429 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-08-30 15:48:59.068984478 +0000 2024-08-30T15:57:28.429 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-08-30 15:48:59.068984478 +0000 2024-08-30T15:57:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-08-30 15:48:59.068984478 +0000 2024-08-30T15:57:28.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-30T15:57:28.492 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-08-30T15:57:28.492 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-08-30T15:57:28.492 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 8.7428e-05 s, 5.9 MB/s 2024-08-30T15:57:28.493 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-30T15:57:28.548 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2024-08-30T15:57:28.604 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-30T15:57:28.604 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T15:57:28.604 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-08-30T15:57:28.604 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T15:57:28.604 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T15:57:28.604 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-08-30 15:51:24.812321468 +0000 2024-08-30T15:57:28.604 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-08-30 15:48:59.397974354 +0000 2024-08-30T15:57:28.605 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-08-30 15:48:59.397974354 +0000 2024-08-30T15:57:28.605 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-08-30 15:48:59.397974354 +0000 2024-08-30T15:57:28.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-30T15:57:28.667 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-08-30T15:57:28.667 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-08-30T15:57:28.667 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.5124e-05 s, 5.4 MB/s 2024-08-30T15:57:28.669 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-30T15:57:28.724 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-30T15:57:28.725 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:57:28.725 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2024-08-30T15:57:28.744 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-30T15:57:28.744 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 2024-08-30T15:57:28.802 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-30T15:57:28.802 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T15:57:28.802 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-08-30T15:57:28.802 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T15:57:28.802 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T15:57:28.803 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-08-30 15:51:18.438567956 +0000 2024-08-30T15:57:28.803 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-08-30 15:48:57.942121662 +0000 2024-08-30T15:57:28.803 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-08-30 15:48:57.942121662 +0000 2024-08-30T15:57:28.803 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-08-30 15:48:57.942121662 +0000 2024-08-30T15:57:28.803 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-30T15:57:28.871 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-08-30T15:57:28.871 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-08-30T15:57:28.871 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000119844 s, 4.3 MB/s 2024-08-30T15:57:28.872 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-30T15:57:28.932 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 2024-08-30T15:57:28.990 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-30T15:57:28.990 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T15:57:28.990 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-08-30T15:57:28.990 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T15:57:28.990 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T15:57:28.990 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-08-30 15:51:18.439567938 +0000 2024-08-30T15:57:28.991 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-08-30 15:48:58.250116844 +0000 2024-08-30T15:57:28.991 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-08-30 15:48:58.250116844 +0000 2024-08-30T15:57:28.991 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-08-30 15:48:58.250116844 +0000 2024-08-30T15:57:28.991 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-30T15:57:29.056 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-08-30T15:57:29.057 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-08-30T15:57:29.057 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000118222 s, 4.3 MB/s 2024-08-30T15:57:29.058 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-30T15:57:29.117 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 2024-08-30T15:57:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-30T15:57:29.173 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T15:57:29.173 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-08-30T15:57:29.173 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T15:57:29.173 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T15:57:29.173 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-08-30 15:51:18.439567938 +0000 2024-08-30T15:57:29.174 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-08-30 15:48:58.551112134 +0000 2024-08-30T15:57:29.174 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-08-30 15:48:58.551112134 +0000 2024-08-30T15:57:29.174 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-08-30 15:48:58.551112134 +0000 2024-08-30T15:57:29.174 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-30T15:57:29.239 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-08-30T15:57:29.239 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-08-30T15:57:29.239 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000107284 s, 4.8 MB/s 2024-08-30T15:57:29.240 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-30T15:57:29.301 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 2024-08-30T15:57:29.357 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-30T15:57:29.358 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T15:57:29.358 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-08-30T15:57:29.358 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T15:57:29.358 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T15:57:29.358 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-08-30 15:51:18.439567938 +0000 2024-08-30T15:57:29.358 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-08-30 15:48:58.873107095 +0000 2024-08-30T15:57:29.358 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-08-30 15:48:58.873107095 +0000 2024-08-30T15:57:29.358 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-08-30 15:48:58.873107095 +0000 2024-08-30T15:57:29.359 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-30T15:57:29.422 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-08-30T15:57:29.422 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-08-30T15:57:29.422 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000131376 s, 3.9 MB/s 2024-08-30T15:57:29.423 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-30T15:57:29.480 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-30T15:57:29.480 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-08-30T15:57:29.480 INFO:tasks.ceph:Generating config... 2024-08-30T15:57:29.481 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-08-30T15:57:29.482 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-08-30T15:57:29.482 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-08-30T15:57:29.482 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-08-30T15:57:29.482 INFO:tasks.ceph:[global] ms type = async 2024-08-30T15:57:29.482 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-08-30T15:57:29.482 INFO:tasks.ceph:[global] osd op thread timeout = 150 2024-08-30T15:57:29.482 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-30T15:57:29.482 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-30T15:57:29.482 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-30T15:57:29.482 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-30T15:57:29.482 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-30T15:57:29.482 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-08-30T15:57:29.482 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-08-30T15:57:29.483 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-08-30T15:57:29.483 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-08-30T15:57:29.483 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-08-30T15:57:29.483 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-08-30T15:57:29.483 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-08-30T15:57:29.483 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-08-30T15:57:29.483 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-08-30T15:57:29.483 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-08-30T15:57:29.483 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-08-30T15:57:29.483 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-08-30T15:57:29.483 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-08-30T15:57:29.483 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-08-30T15:57:29.484 INFO:tasks.ceph:[osd] debug monc = 20 2024-08-30T15:57:29.484 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-30T15:57:29.484 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-30T15:57:29.484 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-08-30T15:57:29.485 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-08-30T15:57:29.485 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-08-30T15:57:29.485 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-08-30T15:57:29.485 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-08-30T15:57:29.485 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-08-30T15:57:29.485 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-08-30T15:57:29.485 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-08-30T15:57:29.485 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-08-30T15:57:29.485 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-08-30T15:57:29.485 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-08-30T15:57:29.485 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-08-30T15:57:29.485 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-08-30T15:57:29.485 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-08-30T15:57:29.485 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-08-30T15:57:29.486 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-08-30T15:57:29.486 INFO:tasks.ceph:Setting up mon.a... 2024-08-30T15:57:29.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-08-30T15:57:29.537 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.keyring 2024-08-30T15:57:29.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-08-30T15:57:29.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-30T15:57:29.648 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.50:6789]'), ('mon.c', '[v1:172.21.15.50:6790]'), ('mon.b', '[v1:172.21.15.94:6789]')] 2024-08-30T15:57:29.649 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.50:6789],[v1:172.21.15.50:6790],[v1:172.21.15.94:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80, 'osd op thread timeout': 150}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-08-30T15:57:29.650 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:29.650 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-30T15:57:29.703 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.50:6789]' --addv c '[v1:172.21.15.50:6790]' --addv b '[v1:172.21.15.94:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-08-30T15:57:29.788 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-08-30T15:57:29.789 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: generated fsid 9d72e2e2-0554-42af-8e62-9b1a3a804112 2024-08-30T15:57:29.789 INFO:teuthology.orchestra.run.smithi050.stdout:setting min_mon_release = pacific 2024-08-30T15:57:29.789 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 0 2024-08-30T15:57:29.789 INFO:teuthology.orchestra.run.smithi050.stdout:fsid 9d72e2e2-0554-42af-8e62-9b1a3a804112 2024-08-30T15:57:29.789 INFO:teuthology.orchestra.run.smithi050.stdout:last_changed 2024-08-30T15:57:29.788167+0000 2024-08-30T15:57:29.789 INFO:teuthology.orchestra.run.smithi050.stdout:created 2024-08-30T15:57:29.788167+0000 2024-08-30T15:57:29.789 INFO:teuthology.orchestra.run.smithi050.stdout:min_mon_release 16 (pacific) 2024-08-30T15:57:29.789 INFO:teuthology.orchestra.run.smithi050.stdout:election_strategy: 1 2024-08-30T15:57:29.789 INFO:teuthology.orchestra.run.smithi050.stdout:0: v1:172.21.15.50:6789/0 mon.a 2024-08-30T15:57:29.789 INFO:teuthology.orchestra.run.smithi050.stdout:1: v1:172.21.15.94:6789/0 mon.b 2024-08-30T15:57:29.789 INFO:teuthology.orchestra.run.smithi050.stdout:2: v1:172.21.15.50:6790/0 mon.c 2024-08-30T15:57:29.790 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-08-30T15:57:29.791 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-30T15:57:29.805 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9d72e2e2-0554-42af-8e62-9b1a3a804112... 2024-08-30T15:57:29.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-30T15:57:29.848 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-30T15:57:29.888 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2024-08-30T15:57:29.888 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2024-08-30T15:57:29.888 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-30T15:57:29.888 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2024-08-30T15:57:29.889 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.889 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2024-08-30T15:57:29.889 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.889 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2024-08-30T15:57:29.889 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.889 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2024-08-30T15:57:29.889 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2024-08-30T15:57:29.889 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.889 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2024-08-30T15:57:29.889 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2024-08-30T15:57:29.889 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.889 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-30T15:57:29.889 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2024-08-30T15:57:29.890 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.890 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2024-08-30T15:57:29.890 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-30T15:57:29.890 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.890 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2024-08-30T15:57:29.890 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.890 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2024-08-30T15:57:29.890 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2024-08-30T15:57:29.890 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2024-08-30T15:57:29.890 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2024-08-30T15:57:29.890 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2024-08-30T15:57:29.891 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2024-08-30T15:57:29.891 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2024-08-30T15:57:29.891 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-30T15:57:29.891 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-30T15:57:29.891 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2024-08-30T15:57:29.891 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.891 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-30T15:57:29.891 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.891 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2024-08-30T15:57:29.891 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.891 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2024-08-30T15:57:29.891 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.891 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2024-08-30T15:57:29.891 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2024-08-30T15:57:29.892 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2024-08-30T15:57:29.892 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = "[v1:172.21.15.50:6789],[v1:172.21.15.50:6790],[v1:172.21.15.94:6789]" 2024-08-30T15:57:29.892 INFO:teuthology.orchestra.run.smithi050.stdout: mon client directed command retry = 5 2024-08-30T15:57:29.892 INFO:teuthology.orchestra.run.smithi050.stdout: mon election default strategy = 1 2024-08-30T15:57:29.892 INFO:teuthology.orchestra.run.smithi050.stdout: ms bind msgr2 = False 2024-08-30T15:57:29.892 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject socket failures = 5000 2024-08-30T15:57:29.892 INFO:teuthology.orchestra.run.smithi050.stdout: ms type = async 2024-08-30T15:57:29.892 INFO:teuthology.orchestra.run.smithi050.stdout: osd heartbeat grace = 80 2024-08-30T15:57:29.892 INFO:teuthology.orchestra.run.smithi050.stdout: osd op thread timeout = 150 2024-08-30T15:57:29.892 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 9d72e2e2-0554-42af-8e62-9b1a3a804112 2024-08-30T15:57:29.892 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.892 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2024-08-30T15:57:29.893 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2024-08-30T15:57:29.893 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.893 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2024-08-30T15:57:29.893 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 600 2024-08-30T15:57:29.893 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock profile = high_recovery_ops 2024-08-30T15:57:29.893 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.893 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2024-08-30T15:57:29.893 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2024-08-30T15:57:29.893 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.893 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2024-08-30T15:57:29.893 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2024-08-30T15:57:29.893 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2024-08-30T15:57:29.893 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.894 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2024-08-30T15:57:29.894 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2024-08-30T15:57:29.894 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.894 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2024-08-30T15:57:29.894 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.894 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2024-08-30T15:57:29.894 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.894 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2024-08-30T15:57:29.894 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.894 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2024-08-30T15:57:29.894 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2024-08-30T15:57:29.895 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.895 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2024-08-30T15:57:29.895 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2024-08-30T15:57:29.895 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore block size = 96636764160 2024-08-30T15:57:29.895 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore compression algorithm = zstd 2024-08-30T15:57:29.895 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore compression mode = aggressive 2024-08-30T15:57:29.895 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore fsck on mount = True 2024-08-30T15:57:29.895 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore zero block detection = True 2024-08-30T15:57:29.895 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluefs = 1 2024-08-30T15:57:29.895 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluestore = 1 2024-08-30T15:57:29.895 INFO:teuthology.orchestra.run.smithi050.stdout: debug monc = 20 2024-08-30T15:57:29.895 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-08-30T15:57:29.895 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 20 2024-08-30T15:57:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: debug rocksdb = 10 2024-08-30T15:57:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd backfillfull_ratio = 0.85 2024-08-30T15:57:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd full ratio = 0.9 2024-08-30T15:57:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd nearfull ratio = 0.8 2024-08-30T15:57:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: osd blocked scrub grace period = 3600 2024-08-30T15:57:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify cached snaps = True 2024-08-30T15:57:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify missing on start = True 2024-08-30T15:57:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: osd failsafe full ratio = 0.95 2024-08-30T15:57:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: osd fast shutdown = False 2024-08-30T15:57:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: osd max markdown count = 1000 2024-08-30T15:57:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock override recovery settings = True 2024-08-30T15:57:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: osd objectstore = bluestore 2024-08-30T15:57:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: osd op queue = debug_random 2024-08-30T15:57:29.896 INFO:teuthology.orchestra.run.smithi050.stdout: osd op queue cut off = debug_random 2024-08-30T15:57:29.897 INFO:teuthology.orchestra.run.smithi050.stdout: osd shutdown pgref assert = True 2024-08-30T15:57:29.897 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.897 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2024-08-30T15:57:29.897 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-08-30T15:57:29.897 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2024-08-30T15:57:29.897 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2024-08-30T15:57:29.897 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2024-08-30T15:57:29.897 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2024-08-30T15:57:29.897 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2024-08-30T15:57:29.897 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2024-08-30T15:57:29.897 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.897 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2024-08-30T15:57:29.897 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-08-30T15:57:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2024-08-30T15:57:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2024-08-30T15:57:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2024-08-30T15:57:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2024-08-30T15:57:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2024-08-30T15:57:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2024-08-30T15:57:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2024-08-30T15:57:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2024-08-30T15:57:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2024-08-30T15:57:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-30T15:57:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2024-08-30T15:57:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2024-08-30T15:57:29.898 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.899 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2024-08-30T15:57:29.899 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-30T15:57:29.899 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-30T15:57:29.899 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.899 INFO:teuthology.orchestra.run.smithi050.stdout: # 1m isn't quite enough 2024-08-30T15:57:29.899 INFO:teuthology.orchestra.run.smithi050.stdout: mon_down_mkfs_grace = 2m 2024-08-30T15:57:29.899 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.899 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_filestore_osds = false 2024-08-30T15:57:29.899 INFO:teuthology.orchestra.run.smithi050.stdout: mon min osdmap epochs = 50 2024-08-30T15:57:29.899 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd crush smoke test = False 2024-08-30T15:57:29.899 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune interval = 2 2024-08-30T15:57:29.899 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune min = 15 2024-08-30T15:57:29.899 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune txsize = 2 2024-08-30T15:57:29.899 INFO:teuthology.orchestra.run.smithi050.stdout: mon scrub interval = 300 2024-08-30T15:57:29.900 INFO:teuthology.orchestra.run.smithi050.stdout: paxos service trim min = 10 2024-08-30T15:57:29.900 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:57:29.900 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2024-08-30T15:57:29.900 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2024-08-30T15:57:29.900 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2024-08-30T15:57:29.900 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2024-08-30T15:57:29.900 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-30T15:57:29.900 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-30T15:57:29.900 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2024-08-30T15:57:29.900 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.c] 2024-08-30T15:57:29.900 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.b] 2024-08-30T15:57:29.901 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2024-08-30T15:57:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: chdir = "" 2024-08-30T15:57:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-30T15:57:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: auth supported = cephx 2024-08-30T15:57:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: filestore xattr use omap = true 2024-08-30T15:57:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: mon clock drift allowed = 1.000 2024-08-30T15:57:29.901 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: osd crush chooseleaf type = 0 2024-08-30T15:57:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: auth debug = true 2024-08-30T15:57:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on old message = true 2024-08-30T15:57:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on bug = true 2024-08-30T15:57:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-30T15:57:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: mon pg warn max object skew = 0 2024-08-30T15:57:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: # disable pg_autoscaler by default for new pools 2024-08-30T15:57:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-30T15:57:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default size = 2 2024-08-30T15:57:29.902 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow primary affinity = true 2024-08-30T15:57:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow pg remap = true 2024-08-30T15:57:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on legacy crush tunables = false 2024-08-30T15:57:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on crush straw calc version zero = false 2024-08-30T15:57:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on no sortbitwise = false 2024-08-30T15:57:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on osd down out interval zero = false 2024-08-30T15:57:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on too few osds = false 2024-08-30T15:57:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-30T15:57:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-30T15:57:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: mon_allow_pool_size_one = true 2024-08-30T15:57:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-30T15:57:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: osd default data pool replay window = 5 2024-08-30T15:57:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: mon allow pool delete = true 2024-08-30T15:57:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: mon cluster log file level = debug 2024-08-30T15:57:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: debug asserts on shutdown = true 2024-08-30T15:57:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: mon health detail to clog = false 2024-08-30T15:57:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: mon host = "[v1:172.21.15.50:6789],[v1:172.21.15.50:6790],[v1:172.21.15.94:6789]" 2024-08-30T15:57:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: mon client directed command retry = 5 2024-08-30T15:57:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: mon election default strategy = 1 2024-08-30T15:57:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: ms bind msgr2 = False 2024-08-30T15:57:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: ms inject socket failures = 5000 2024-08-30T15:57:29.904 INFO:teuthology.orchestra.run.smithi094.stdout: ms type = async 2024-08-30T15:57:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: osd heartbeat grace = 80 2024-08-30T15:57:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: osd op thread timeout = 150 2024-08-30T15:57:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = 9d72e2e2-0554-42af-8e62-9b1a3a804112 2024-08-30T15:57:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.905 INFO:teuthology.orchestra.run.smithi094.stdout:[osd] 2024-08-30T15:57:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: osd journal size = 100 2024-08-30T15:57:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub load threshold = 5.0 2024-08-30T15:57:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub max interval = 600 2024-08-30T15:57:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: osd mclock profile = high_recovery_ops 2024-08-30T15:57:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: osd recover clone overlap = true 2024-08-30T15:57:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: osd recovery max chunk = 1048576 2024-08-30T15:57:29.905 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug shutdown = true 2024-08-30T15:57:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug op order = true 2024-08-30T15:57:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify stray on activate = true 2024-08-30T15:57:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: osd open classes on start = true 2024-08-30T15:57:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug pg log writeout = true 2024-08-30T15:57:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: osd deep scrub update digest min age = 30 2024-08-30T15:57:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: osd map max advance = 10 2024-08-30T15:57:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: journal zero on create = true 2024-08-30T15:57:29.906 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: filestore ondisk finisher threads = 3 2024-08-30T15:57:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: filestore apply finisher threads = 3 2024-08-30T15:57:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: bdev debug aio = true 2024-08-30T15:57:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug misdirected ops = true 2024-08-30T15:57:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore block size = 96636764160 2024-08-30T15:57:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore compression algorithm = zstd 2024-08-30T15:57:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore compression mode = aggressive 2024-08-30T15:57:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore fsck on mount = True 2024-08-30T15:57:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore zero block detection = True 2024-08-30T15:57:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluefs = 1 2024-08-30T15:57:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluestore = 1 2024-08-30T15:57:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: debug monc = 20 2024-08-30T15:57:29.907 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-08-30T15:57:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: debug osd = 20 2024-08-30T15:57:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: debug rocksdb = 10 2024-08-30T15:57:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd backfillfull_ratio = 0.85 2024-08-30T15:57:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd full ratio = 0.9 2024-08-30T15:57:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd nearfull ratio = 0.8 2024-08-30T15:57:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: osd blocked scrub grace period = 3600 2024-08-30T15:57:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify cached snaps = True 2024-08-30T15:57:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify missing on start = True 2024-08-30T15:57:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: osd failsafe full ratio = 0.95 2024-08-30T15:57:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: osd fast shutdown = False 2024-08-30T15:57:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: osd max markdown count = 1000 2024-08-30T15:57:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: osd mclock override recovery settings = True 2024-08-30T15:57:29.908 INFO:teuthology.orchestra.run.smithi094.stdout: osd objectstore = bluestore 2024-08-30T15:57:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: osd op queue = debug_random 2024-08-30T15:57:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: osd op queue cut off = debug_random 2024-08-30T15:57:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: osd shutdown pgref assert = True 2024-08-30T15:57:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.909 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr] 2024-08-30T15:57:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-08-30T15:57:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgr = 20 2024-08-30T15:57:29.909 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2024-08-30T15:57:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2024-08-30T15:57:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2024-08-30T15:57:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2024-08-30T15:57:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: mgr/telemetry/nag = false 2024-08-30T15:57:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.912 INFO:teuthology.orchestra.run.smithi094.stdout:[mon] 2024-08-30T15:57:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-08-30T15:57:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2024-08-30T15:57:29.912 INFO:teuthology.orchestra.run.smithi094.stdout: debug paxos = 20 2024-08-30T15:57:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2024-08-30T15:57:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: mon data avail warn = 5 2024-08-30T15:57:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: mon mgr mkfs grace = 240 2024-08-30T15:57:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2024-08-30T15:57:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd reporter subtree level = osd 2024-08-30T15:57:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd prime pg temp = true 2024-08-30T15:57:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2024-08-30T15:57:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-30T15:57:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: auth mon ticket ttl = 660 # 11m 2024-08-30T15:57:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: auth service ticket ttl = 240 # 4m 2024-08-30T15:57:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.913 INFO:teuthology.orchestra.run.smithi094.stdout: # don't complain about insecure global_id in the test suite 2024-08-30T15:57:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-30T15:57:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-30T15:57:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: # 1m isn't quite enough 2024-08-30T15:57:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: mon_down_mkfs_grace = 2m 2024-08-30T15:57:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_filestore_osds = false 2024-08-30T15:57:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: mon min osdmap epochs = 50 2024-08-30T15:57:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd crush smoke test = False 2024-08-30T15:57:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune interval = 2 2024-08-30T15:57:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune min = 15 2024-08-30T15:57:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune txsize = 2 2024-08-30T15:57:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: mon scrub interval = 300 2024-08-30T15:57:29.914 INFO:teuthology.orchestra.run.smithi094.stdout: paxos service trim min = 10 2024-08-30T15:57:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-08-30T15:57:29.915 INFO:teuthology.orchestra.run.smithi094.stdout:[client] 2024-08-30T15:57:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: rgw cache enabled = true 2024-08-30T15:57:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable ops log = true 2024-08-30T15:57:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable usage log = true 2024-08-30T15:57:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-30T15:57:29.915 INFO:teuthology.orchestra.run.smithi094.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-30T15:57:29.915 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.a] 2024-08-30T15:57:29.915 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.c] 2024-08-30T15:57:29.915 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.b] 2024-08-30T15:57:29.917 INFO:tasks.ceph:Creating admin key on mon.a... 2024-08-30T15:57:29.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-08-30T15:57:30.009 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-30T15:57:30.009 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:30.010 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-08-30T15:57:30.025 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:30.025 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-08-30T15:57:30.079 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2024-08-30T15:57:30.080 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:30.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-30T15:57:30.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-30T15:57:30.154 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:30.154 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-30T15:57:30.209 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi094.front.sepia.ceph.com 2024-08-30T15:57:30.209 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:57:30.209 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-30T15:57:30.209 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-30T15:57:30.247 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:57:30.247 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-30T15:57:30.303 INFO:tasks.ceph:Setting up mon nodes... 2024-08-30T15:57:30.303 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-30T15:57:30.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-08-30T15:57:30.353 INFO:teuthology.orchestra.run.smithi050.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-08-30T15:57:30.356 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-08-30T15:57:30.417 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-08-30T15:57:30.419 INFO:tasks.ceph:Setting up mds nodes... 2024-08-30T15:57:30.419 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-30T15:57:30.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-30T15:57:30.459 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-08-30T15:57:30.472 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-08-30T15:57:30.514 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-08-30T15:57:30.529 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-30T15:57:30.529 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-08-30T15:57:30.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-08-30T15:57:30.554 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-30T15:57:30.554 INFO:tasks.ceph:role: osd.0 2024-08-30T15:57:30.554 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com 2024-08-30T15:57:30.555 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-30T15:57:30.618 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-30T15:57:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-30T15:57:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-30T15:57:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-30T15:57:30.619 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-30T15:57:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-08-30T15:57:30.619 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-30T15:57:30.619 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-30T15:57:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-30T15:57:30.619 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-30T15:57:30.651 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-08-30T15:57:30.653 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-08-30T15:57:30.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-08-30T15:57:30.722 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-08-30T15:57:30.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-08-30T15:57:30.860 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-30T15:57:30.860 INFO:tasks.ceph:role: osd.1 2024-08-30T15:57:30.860 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com 2024-08-30T15:57:30.860 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-30T15:57:30.927 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-30T15:57:30.927 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-30T15:57:30.927 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-30T15:57:30.927 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-30T15:57:30.927 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-30T15:57:30.927 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-08-30T15:57:30.927 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-30T15:57:30.927 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-30T15:57:30.927 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-30T15:57:30.927 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-30T15:57:30.959 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-08-30T15:57:30.961 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-08-30T15:57:30.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-08-30T15:57:31.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-08-30T15:57:31.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-08-30T15:57:31.169 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-30T15:57:31.169 INFO:tasks.ceph:role: osd.2 2024-08-30T15:57:31.169 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com 2024-08-30T15:57:31.169 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-30T15:57:31.233 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-30T15:57:31.233 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-30T15:57:31.233 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-30T15:57:31.233 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-30T15:57:31.233 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-30T15:57:31.233 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-08-30T15:57:31.233 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-30T15:57:31.234 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-30T15:57:31.234 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-30T15:57:31.234 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-30T15:57:31.265 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-08-30T15:57:31.267 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-08-30T15:57:31.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-08-30T15:57:31.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-08-30T15:57:31.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-08-30T15:57:31.470 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-30T15:57:31.471 INFO:tasks.ceph:role: osd.3 2024-08-30T15:57:31.471 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi050.front.sepia.ceph.com 2024-08-30T15:57:31.471 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-08-30T15:57:31.535 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-08-30T15:57:31.535 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-30T15:57:31.535 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-30T15:57:31.535 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-30T15:57:31.535 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-30T15:57:31.535 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-08-30T15:57:31.536 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-30T15:57:31.536 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-30T15:57:31.536 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-30T15:57:31.536 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-30T15:57:31.570 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-08-30T15:57:31.572 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-08-30T15:57:31.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-08-30T15:57:31.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-08-30T15:57:31.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-30T15:57:31.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:31.798+0000 7fd8f2e16640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-08-30T15:57:31.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:31.798+0000 7fd8f2e16640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-08-30T15:57:31.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:31.798+0000 7fd8f2e16640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-08-30T15:57:31.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:31.798+0000 7fd8f2e16640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-08-30T15:57:31.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:31.798+0000 7fd8f2e16640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-08-30T15:57:31.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:31.798+0000 7fd8f2e16640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-08-30T15:57:34.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-30T15:57:34.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-30T15:57:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:34.760+0000 7f245e413640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-08-30T15:57:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:34.760+0000 7f245e413640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-08-30T15:57:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:34.760+0000 7f245e413640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-08-30T15:57:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:34.760+0000 7f245e413640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-08-30T15:57:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:34.760+0000 7f245e413640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-08-30T15:57:34.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:34.761+0000 7f245e413640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-08-30T15:57:37.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-30T15:57:37.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-30T15:57:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:37.716+0000 7f7bf2a18640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-08-30T15:57:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:37.717+0000 7f7bf2a18640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-08-30T15:57:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:37.717+0000 7f7bf2a18640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-08-30T15:57:37.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:37.717+0000 7f7bf2a18640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-08-30T15:57:37.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:37.717+0000 7f7bf2a18640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-08-30T15:57:37.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:37.717+0000 7f7bf2a18640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-08-30T15:57:40.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-30T15:57:40.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-30T15:57:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:40.680+0000 7fda15864640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-08-30T15:57:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:40.680+0000 7fda15864640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-08-30T15:57:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:40.680+0000 7fda15864640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-08-30T15:57:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:40.680+0000 7fda15864640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-08-30T15:57:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:40.680+0000 7fda15864640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-08-30T15:57:40.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-08-30T15:57:40.681+0000 7fda15864640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-08-30T15:57:43.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-30T15:57:43.553 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-08-30T15:57:43.583 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-30T15:57:43.583 INFO:tasks.ceph:role: osd.4 2024-08-30T15:57:43.583 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com 2024-08-30T15:57:43.583 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-30T15:57:43.652 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-30T15:57:43.653 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-30T15:57:43.653 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-30T15:57:43.653 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-30T15:57:43.653 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-30T15:57:43.653 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-08-30T15:57:43.653 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-30T15:57:43.653 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-30T15:57:43.653 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-30T15:57:43.653 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-30T15:57:43.690 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-08-30T15:57:43.693 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-08-30T15:57:43.693 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-08-30T15:57:43.764 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-08-30T15:57:43.834 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-08-30T15:57:43.899 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-30T15:57:43.900 INFO:tasks.ceph:role: osd.5 2024-08-30T15:57:43.900 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com 2024-08-30T15:57:43.900 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-30T15:57:43.963 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-30T15:57:43.964 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-30T15:57:43.964 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-30T15:57:43.964 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-30T15:57:43.964 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-30T15:57:43.964 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-08-30T15:57:43.964 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-30T15:57:43.964 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-30T15:57:43.964 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-30T15:57:43.964 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-30T15:57:43.998 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-08-30T15:57:44.000 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-08-30T15:57:44.000 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-08-30T15:57:44.069 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-08-30T15:57:44.138 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-08-30T15:57:44.203 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-30T15:57:44.203 INFO:tasks.ceph:role: osd.6 2024-08-30T15:57:44.203 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com 2024-08-30T15:57:44.203 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-30T15:57:44.268 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-30T15:57:44.268 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-30T15:57:44.268 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-30T15:57:44.269 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-30T15:57:44.269 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-30T15:57:44.269 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-08-30T15:57:44.269 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-30T15:57:44.269 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-30T15:57:44.269 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-30T15:57:44.269 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-30T15:57:44.305 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-08-30T15:57:44.307 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-08-30T15:57:44.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-08-30T15:57:44.377 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-08-30T15:57:44.446 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-08-30T15:57:44.511 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-30T15:57:44.511 INFO:tasks.ceph:role: osd.7 2024-08-30T15:57:44.511 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi094.front.sepia.ceph.com 2024-08-30T15:57:44.511 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-08-30T15:57:44.576 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-08-30T15:57:44.576 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-30T15:57:44.576 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-30T15:57:44.576 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-30T15:57:44.576 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-30T15:57:44.576 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-08-30T15:57:44.576 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-30T15:57:44.576 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-30T15:57:44.576 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-30T15:57:44.576 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-30T15:57:44.610 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-08-30T15:57:44.612 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-08-30T15:57:44.612 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-08-30T15:57:44.680 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-08-30T15:57:44.749 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-30T15:57:44.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:44.844+0000 7f5e6081c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-08-30T15:57:44.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:44.844+0000 7f5e6081c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-08-30T15:57:44.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:44.844+0000 7f5e6081c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-08-30T15:57:44.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:44.844+0000 7f5e6081c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-08-30T15:57:44.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:44.844+0000 7f5e6081c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-08-30T15:57:44.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:44.844+0000 7f5e6081c640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-08-30T15:57:47.684 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-08-30T15:57:47.713 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-30T15:57:47.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:47.801+0000 7f9c6a085640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-08-30T15:57:47.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:47.801+0000 7f9c6a085640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-08-30T15:57:47.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:47.801+0000 7f9c6a085640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-08-30T15:57:47.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:47.801+0000 7f9c6a085640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-08-30T15:57:47.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:47.801+0000 7f9c6a085640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-08-30T15:57:47.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:47.802+0000 7f9c6a085640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-08-30T15:57:50.644 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-08-30T15:57:50.673 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-30T15:57:50.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:50.762+0000 7f4b0260e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-08-30T15:57:50.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:50.762+0000 7f4b0260e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-08-30T15:57:50.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:50.762+0000 7f4b0260e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-08-30T15:57:50.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:50.762+0000 7f4b0260e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-08-30T15:57:50.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:50.762+0000 7f4b0260e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-08-30T15:57:50.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:50.763+0000 7f4b0260e640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-08-30T15:57:53.611 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-08-30T15:57:53.641 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-30T15:57:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:53.730+0000 7f93dae0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-08-30T15:57:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:53.730+0000 7f93dae0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-08-30T15:57:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:53.730+0000 7f93dae0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-08-30T15:57:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:53.730+0000 7f93dae0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-08-30T15:57:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:53.730+0000 7f93dae0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-08-30T15:57:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-08-30T15:57:53.730+0000 7f93dae0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-08-30T15:57:56.568 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-08-30T15:57:56.596 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-30T15:57:56.596 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:56.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-08-30T15:57:56.626 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:56.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-08-30T15:57:56.694 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:56.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-08-30T15:57:56.761 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:56.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-08-30T15:57:56.827 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:56.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-08-30T15:57:56.895 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:57:56.895 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-08-30T15:57:56.926 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:57:56.926 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-08-30T15:57:56.996 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:57:56.996 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-08-30T15:57:57.064 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:57:57.064 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-08-30T15:57:57.131 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:57:57.131 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-08-30T15:57:57.198 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:57:57.198 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-08-30T15:57:57.216 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:57:57.216 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-08-30T15:57:57.254 INFO:tasks.ceph:Adding keys to all mons... 2024-08-30T15:57:57.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-30T15:57:57.258 DEBUG:teuthology.orchestra.run.smithi094:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-30T15:57:57.297 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.y] 2024-08-30T15:57:57.298 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDq69FmKTwRFRAAzHeK7CgAef5+ZrDEWb8j9Q== 2024-08-30T15:57:57.298 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2024-08-30T15:57:57.298 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDr69FmJjWiLxAAWrqLJ+RAvZcro+Y8w2R1Rw== 2024-08-30T15:57:57.298 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2024-08-30T15:57:57.298 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDu69FmyE5iLRAAbaq2eh1h0JEt01gqTsNelA== 2024-08-30T15:57:57.298 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2024-08-30T15:57:57.298 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDx69FmKtbVKhAAiLppgZFaT9nqjObwEuUAIw== 2024-08-30T15:57:57.298 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.3] 2024-08-30T15:57:57.298 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQD069FmtImrKBAAlOoy7rjKkOkbKiMCGz/pzA== 2024-08-30T15:57:57.298 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.x] 2024-08-30T15:57:57.298 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDq69FmWz3UGBAA7isB+adSr+K6snKpfgDGug== 2024-08-30T15:57:57.298 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.4] 2024-08-30T15:57:57.299 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQD469FmMWxRMhAA+UE6QP4E3DhyqyWe5CrDOw== 2024-08-30T15:57:57.299 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.5] 2024-08-30T15:57:57.299 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQD769FmrZHYLxAAgTxqUqEdvVDolsxrNhS0gw== 2024-08-30T15:57:57.299 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.6] 2024-08-30T15:57:57.299 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQD+69FmCkCCLRAAhbd3lKik/WswNtjL9xdYrQ== 2024-08-30T15:57:57.299 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.7] 2024-08-30T15:57:57.299 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAB7NFmxv2YKxAA1Yq5mKdpH8nsbo3cul85oQ== 2024-08-30T15:57:57.299 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2024-08-30T15:57:57.299 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDq69FmbNdhGxAAZz5CMAq8Gg24gkPoiWvFNA== 2024-08-30T15:57:57.299 INFO:teuthology.orchestra.run.smithi050.stdout:[client.1] 2024-08-30T15:57:57.299 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDq69Fm15+fHhAAK5dQS6OzLDIhYXZXfKRHsA== 2024-08-30T15:57:57.318 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.y] 2024-08-30T15:57:57.318 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDq69FmKTwRFRAAzHeK7CgAef5+ZrDEWb8j9Q== 2024-08-30T15:57:57.318 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2024-08-30T15:57:57.318 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDr69FmJjWiLxAAWrqLJ+RAvZcro+Y8w2R1Rw== 2024-08-30T15:57:57.318 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2024-08-30T15:57:57.319 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDu69FmyE5iLRAAbaq2eh1h0JEt01gqTsNelA== 2024-08-30T15:57:57.319 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2024-08-30T15:57:57.319 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDx69FmKtbVKhAAiLppgZFaT9nqjObwEuUAIw== 2024-08-30T15:57:57.319 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.3] 2024-08-30T15:57:57.319 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQD069FmtImrKBAAlOoy7rjKkOkbKiMCGz/pzA== 2024-08-30T15:57:57.319 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.x] 2024-08-30T15:57:57.319 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDq69FmWz3UGBAA7isB+adSr+K6snKpfgDGug== 2024-08-30T15:57:57.319 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.4] 2024-08-30T15:57:57.319 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQD469FmMWxRMhAA+UE6QP4E3DhyqyWe5CrDOw== 2024-08-30T15:57:57.319 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.5] 2024-08-30T15:57:57.319 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQD769FmrZHYLxAAgTxqUqEdvVDolsxrNhS0gw== 2024-08-30T15:57:57.319 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.6] 2024-08-30T15:57:57.319 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQD+69FmCkCCLRAAhbd3lKik/WswNtjL9xdYrQ== 2024-08-30T15:57:57.320 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.7] 2024-08-30T15:57:57.320 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAB7NFmxv2YKxAA1Yq5mKdpH8nsbo3cul85oQ== 2024-08-30T15:57:57.320 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-08-30T15:57:57.320 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDq69FmbNdhGxAAZz5CMAq8Gg24gkPoiWvFNA== 2024-08-30T15:57:57.320 INFO:teuthology.orchestra.run.smithi094.stdout:[client.1] 2024-08-30T15:57:57.320 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQDq69Fm15+fHhAAK5dQS6OzLDIhYXZXfKRHsA== 2024-08-30T15:57:57.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-30T15:57:57.343 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-30T15:57:57.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.415 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.468 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.521 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.612 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-30T15:57:57.665 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-30T15:57:57.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.716 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.769 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.873 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-30T15:57:57.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-30T15:57:57.926 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-30T15:57:57.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-30T15:57:57.982 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-30T15:57:58.034 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-30T15:57:58.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-08-30T15:57:58.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-30T15:57:58.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-08-30T15:57:58.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-08-30T15:57:58.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-30T15:57:58.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-08-30T15:57:58.361 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-08-30T15:57:58.386 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-30T15:57:58.481 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-08-30T15:57:58.506 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-30T15:57:58.509 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-30T15:57:58.562 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-08-30T15:57:58.562 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-08-30T15:57:58.563 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-30T15:57:58.563 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-08-30T15:57:58.565 INFO:tasks.ceph.mon.a:Started 2024-08-30T15:57:58.565 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-08-30T15:57:58.565 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-30T15:57:58.566 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-08-30T15:57:58.567 INFO:tasks.ceph.mon.c:Started 2024-08-30T15:57:58.568 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-08-30T15:57:58.568 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-30T15:57:58.569 DEBUG:teuthology.orchestra.run.smithi094:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-08-30T15:57:58.605 INFO:tasks.ceph.mon.b:Started 2024-08-30T15:57:58.605 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-08-30T15:57:58.605 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-08-30T15:57:58.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-08-30T15:57:58.610 INFO:tasks.ceph.mgr.y:Started 2024-08-30T15:57:58.610 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-08-30T15:57:58.610 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-08-30T15:57:58.612 INFO:tasks.ceph.mgr.x:Started 2024-08-30T15:57:58.612 INFO:tasks.ceph:Setting crush tunables to default 2024-08-30T15:57:58.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd crush tunables default 2024-08-30T15:58:05.591 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:06.863 22555== Warning: unimplemented fcntl command: 1036 2024-08-30T15:58:05.900 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:07.173 22555== Warning: unimplemented fcntl command: 1036 2024-08-30T15:58:06.224 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:07.551 22554== Warning: unimplemented fcntl command: 1036 2024-08-30T15:58:06.224 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:07.551 22547== Warning: unimplemented fcntl command: 1036 2024-08-30T15:58:06.552 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:07.879 22547== Warning: unimplemented fcntl command: 1036 2024-08-30T15:58:06.552 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:07.879 22554== Warning: unimplemented fcntl command: 1036 2024-08-30T15:58:21.588 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:21.586+0000 7f8071c0d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-30T15:58:21.619 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:21.617+0000 7fd0b5528040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-30T15:58:21.629 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:21.627+0000 7f8071c0d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-30T15:58:21.664 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:21.662+0000 7fd0b5528040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-30T15:58:21.702 INFO:teuthology.orchestra.run.smithi050.stderr:adjusted tunables profile to default 2024-08-30T15:58:21.718 INFO:tasks.ceph:check_enable_crimson: False 2024-08-30T15:58:21.718 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-08-30T15:58:21.718 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:58:21.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-08-30T15:58:21.744 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:58:21.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-08-30T15:58:21.777 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:21.775+0000 7f8071c0d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-30T15:58:21.810 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:58:21.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-08-30T15:58:21.819 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:21.816+0000 7fd0b5528040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-30T15:58:21.827 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:21.826+0000 7f8071c0d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-30T15:58:21.844 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-08-30T15:58:21.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-08-30T15:58:21.876 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:21.874+0000 7fd0b5528040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-30T15:58:21.901 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:58:21.901 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-08-30T15:58:21.927 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:58:21.927 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-08-30T15:58:21.992 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:58:21.992 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-08-30T15:58:21.993 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:21.992+0000 7f8071c0d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-30T15:58:22.018 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-08-30T15:58:22.019 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-08-30T15:58:22.055 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:22.053+0000 7fd0b5528040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-30T15:58:22.083 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4 0 2024-08-30T15:58:22.110 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:22.108+0000 7f8071c0d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-30T15:58:22.174 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:22.172+0000 7fd0b5528040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-30T15:58:22.416 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:22.415+0000 7f8071c0d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-30T15:58:22.464 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:22.463+0000 7f8071c0d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-30T15:58:22.498 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:22.496+0000 7fd0b5528040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-30T15:58:22.517 INFO:teuthology.orchestra.run.smithi094.stdout:0 2024-08-30T15:58:22.528 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 0fe11a1c-4326-4bfd-a2a4-bff14023f301 1 2024-08-30T15:58:22.556 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:22.554+0000 7fd0b5528040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-30T15:58:22.619 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:22.617+0000 7f8071c0d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-30T15:58:22.711 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:22.709+0000 7fd0b5528040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-30T15:58:23.087 INFO:teuthology.orchestra.run.smithi094.stdout:1 2024-08-30T15:58:23.098 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new a9d5354b-3600-41ed-80c5-a3439da7aca6 2 2024-08-30T15:58:23.570 INFO:teuthology.orchestra.run.smithi094.stdout:2 2024-08-30T15:58:23.575 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:23.574+0000 7f8071c0d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-30T15:58:23.581 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 78bcc398-a7e4-4efc-9836-1dfb94407c9c 3 2024-08-30T15:58:23.630 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:23.629+0000 7f8071c0d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-30T15:58:23.677 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:23.674+0000 7fd0b5528040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-30T15:58:23.737 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:23.735+0000 7fd0b5528040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-30T15:58:23.930 INFO:teuthology.orchestra.run.smithi094.stdout:3 2024-08-30T15:58:23.941 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 53a1060b-7985-4c2c-820e-64b158056113 4 2024-08-30T15:58:24.028 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:24.026+0000 7f8071c0d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-30T15:58:24.116 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:24.114+0000 7f8071c0d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-30T15:58:24.144 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:24.142+0000 7fd0b5528040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-30T15:58:24.229 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:24.227+0000 7f8071c0d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-30T15:58:24.245 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:24.243+0000 7fd0b5528040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-30T15:58:24.277 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:24.275+0000 7f8071c0d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-30T15:58:24.318 INFO:teuthology.orchestra.run.smithi094.stdout:4 2024-08-30T15:58:24.329 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 853a03f7-227d-4f98-9e5c-c29896820746 5 2024-08-30T15:58:24.372 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:24.369+0000 7fd0b5528040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-30T15:58:24.396 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:24.394+0000 7f8071c0d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-30T15:58:24.419 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:24.416+0000 7fd0b5528040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-30T15:58:24.454 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:24.452+0000 7f8071c0d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-30T15:58:24.499 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:24.497+0000 7f8071c0d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-30T15:58:24.540 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:24.537+0000 7fd0b5528040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-30T15:58:24.594 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:24.592+0000 7fd0b5528040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-30T15:58:24.611 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:24.609+0000 7f8071c0d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-30T15:58:24.641 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:24.639+0000 7fd0b5528040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-30T15:58:24.663 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:24.662+0000 7f8071c0d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-30T15:58:24.668 INFO:teuthology.orchestra.run.smithi094.stdout:5 2024-08-30T15:58:24.680 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new bc978060-bf28-4027-8bba-7d6d8659bfc5 6 2024-08-30T15:58:24.718 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:24.716+0000 7f8071c0d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-30T15:58:24.756 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:24.754+0000 7fd0b5528040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-30T15:58:24.788 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:24.787+0000 7f8071c0d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-30T15:58:24.814 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:24.812+0000 7fd0b5528040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-30T15:58:24.834 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:24.832+0000 7f8071c0d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-30T15:58:24.867 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:24.865+0000 7fd0b5528040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-30T15:58:24.879 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:24.877+0000 7f8071c0d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-30T15:58:24.936 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:24.934+0000 7fd0b5528040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-30T15:58:24.988 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:24.986+0000 7fd0b5528040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-30T15:58:25.031 INFO:teuthology.orchestra.run.smithi094.stdout:6 2024-08-30T15:58:25.039 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:25.037+0000 7fd0b5528040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-30T15:58:25.042 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 45df5c6f-b124-46d1-9efb-62b78bc584ed 7 2024-08-30T15:58:25.479 INFO:teuthology.orchestra.run.smithi094.stdout:7 2024-08-30T15:58:25.490 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-30T15:58:25.491 INFO:tasks.ceph.osd.0:Restarting daemon 2024-08-30T15:58:25.491 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-08-30T15:58:25.493 INFO:tasks.ceph.osd.0:Started 2024-08-30T15:58:25.493 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-30T15:58:25.493 INFO:tasks.ceph.osd.1:Restarting daemon 2024-08-30T15:58:25.494 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-08-30T15:58:25.496 INFO:tasks.ceph.osd.1:Started 2024-08-30T15:58:25.496 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-30T15:58:25.496 INFO:tasks.ceph.osd.2:Restarting daemon 2024-08-30T15:58:25.497 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-08-30T15:58:25.499 INFO:tasks.ceph.osd.2:Started 2024-08-30T15:58:25.499 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-30T15:58:25.499 INFO:tasks.ceph.osd.3:Restarting daemon 2024-08-30T15:58:25.499 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-08-30T15:58:25.501 INFO:tasks.ceph.osd.3:Started 2024-08-30T15:58:25.502 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-30T15:58:25.502 INFO:tasks.ceph.osd.4:Restarting daemon 2024-08-30T15:58:25.502 DEBUG:teuthology.orchestra.run.smithi094:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-08-30T15:58:25.504 INFO:tasks.ceph.osd.4:Started 2024-08-30T15:58:25.504 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-30T15:58:25.504 INFO:tasks.ceph.osd.5:Restarting daemon 2024-08-30T15:58:25.504 DEBUG:teuthology.orchestra.run.smithi094:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-08-30T15:58:25.506 INFO:tasks.ceph.osd.5:Started 2024-08-30T15:58:25.506 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-30T15:58:25.507 INFO:tasks.ceph.osd.6:Restarting daemon 2024-08-30T15:58:25.507 DEBUG:teuthology.orchestra.run.smithi094:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-08-30T15:58:25.509 INFO:tasks.ceph.osd.6:Started 2024-08-30T15:58:25.509 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-30T15:58:25.509 INFO:tasks.ceph.osd.7:Restarting daemon 2024-08-30T15:58:25.509 DEBUG:teuthology.orchestra.run.smithi094:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-08-30T15:58:25.511 INFO:tasks.ceph.osd.7:Started 2024-08-30T15:58:25.512 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-30T15:58:25.551 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:25.549+0000 7f8071c0d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-30T15:58:25.808 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-08-30T15:58:25.804+0000 7fd0b5528040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-30T15:58:26.493 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:58:26.493 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":10,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T15:58:25.430425+0000","last_up_change":"0.000000","last_in_change":"2024-08-30T15:58:25.430425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T15:58:26.508 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-08-30T15:58:26.508 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-30T15:58:27.314 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-30T15:58:27.443 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:27.441+0000 7f8012302640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-08-30T15:58:27.631 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T15:58:27.629+0000 7f8012302640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-08-30T15:58:28.435 INFO:teuthology.misc.health.smithi050.stdout: 2024-08-30T15:58:28.436 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T15:58:25.430425+0000","last_up_change":"0.000000","last_in_change":"2024-08-30T15:58:25.430425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T15:58:28.454 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-08-30T15:58:32.422 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:06.849 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:32.494 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:06.921 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:32.499 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:06.918 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:32.559 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:06.978 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:32.692 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:07.122 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:32.741 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:07.151 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:32.751 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:07.182 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:32.799 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:07.210 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:32.910 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:07.336 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:32.972 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:07.399 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:33.042 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:58:33.035+0000 5ef1600 -1 Falling back to public interface 2024-08-30T15:58:33.108 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:58:33.101+0000 5ef1600 -1 Falling back to public interface 2024-08-30T15:58:33.163 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:07.582 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:33.223 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:07.642 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:33.282 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:07.710 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:33.343 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:07.771 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:33.351 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:58:33.343+0000 5ef1600 -1 Falling back to public interface 2024-08-30T15:58:33.374 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:58:33.367+0000 5ef1600 -1 Falling back to public interface 2024-08-30T15:58:33.397 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:07.820 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:33.457 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:07.880 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:33.525 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:58:33.519+0000 5ef1600 -1 Falling back to public interface 2024-08-30T15:58:33.571 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:07.998 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:33.612 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:08.031 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:33.774 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:58:33.767+0000 5ef1600 -1 Falling back to public interface 2024-08-30T15:58:33.796 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:08.226 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:33.823 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:08.235 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:33.930 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:58:33.922+0000 5ef1600 -1 Falling back to public interface 2024-08-30T15:58:33.957 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:08.385 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:33.992 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:08.411 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.022 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:08.449 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.047 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:58:34.040+0000 5ef1600 -1 Falling back to public interface 2024-08-30T15:58:34.174 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:08.605 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.212 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:08.623 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.247 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:08.667 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.290 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:08.718 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.326 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:08.745 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.398 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:08.826 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.424 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:08.851 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.464 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:08.894 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.513 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:08.936 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.540 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:08.951 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.584 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:09.012 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.623 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:09.042 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.625 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:09.045 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.732 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:09.162 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.776 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:09.202 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.794 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:09.223 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.826 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:09.237 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.875 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:09.302 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.911 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:09.334 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.917 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:09.336 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:34.925 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:09.344 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.004 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:09.434 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.064 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:09.491 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.114 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:09.525 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.135 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:09.563 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.166 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:09.593 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.182 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:09.602 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.212 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:09.635 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.213 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:09.632 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.260 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-30T15:58:35.268 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:09.698 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.360 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:09.787 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.413 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:09.824 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.424 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:09.852 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.446 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:09.866 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.455 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:09.883 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.482 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:09.906 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.500 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:09.919 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.539 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:09.969 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.652 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:10.079 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.701 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:10.112 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.717 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:10.136 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.722 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:10.150 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.753 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:10.177 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.757 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:10.185 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.799 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:10.217 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.810 INFO:teuthology.misc.health.smithi050.stdout: 2024-08-30T15:58:35.810 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T15:58:25.430425+0000","last_up_change":"0.000000","last_in_change":"2024-08-30T15:58:25.430425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T15:58:35.814 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:10.244 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.821 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-08-30T15:58:35.945 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:10.372 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.985 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:10.405 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:35.995 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:10.406 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.025 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:10.453 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.029 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:10.453 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.044 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:10.472 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.088 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:10.518 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.093 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:10.512 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.234 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:10.661 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.255 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:10.675 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.286 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:10.698 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.303 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:10.727 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.324 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:10.752 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.522 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:10.942 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.527 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:10.954 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.573 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:10.996 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.590 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:11.017 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.596 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:11.027 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.616 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:11.044 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.622 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:11.049 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.636 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:11.055 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.654 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:11.085 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.668 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:11.087 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.857 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:11.280 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.862 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:11.273 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.894 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:11.305 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:36.909 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:11.337 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:37.020 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:11.439 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:37.052 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:11.471 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:37.084 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:11.511 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:37.116 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:11.543 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:37.407 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:11.830 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:37.439 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:11.862 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:37.480 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:11.908 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:37.540 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:11.969 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:41.897 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:16.327 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:42.196 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:16.615 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:42.431 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:16.850 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:42.520 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:16.946 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:42.627 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-30T15:58:42.697 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:17.108 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:42.835 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:17.263 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:42.919 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:17.342 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:43.061 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:17.489 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:43.391 INFO:teuthology.misc.health.smithi050.stdout: 2024-08-30T15:58:43.392 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T15:58:25.430425+0000","last_up_change":"0.000000","last_in_change":"2024-08-30T15:58:25.430425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T15:58:43.407 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-08-30T15:58:47.806 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:22.224 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:47.810 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:22.229 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:47.961 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:22.391 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:47.967 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:22.398 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:48.545 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:22.972 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:48.550 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:22.976 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:48.558 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:22.969 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:48.564 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:22.975 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:48.732 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:23.155 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:48.736 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:23.160 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:48.806 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:23.225 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:48.810 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:23.230 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:48.857 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:23.286 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:48.861 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:23.290 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:48.941 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:23.368 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:48.948 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:23.375 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:50.214 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-30T15:58:50.868 INFO:teuthology.misc.health.smithi050.stdout: 2024-08-30T15:58:50.868 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T15:58:25.430425+0000","last_up_change":"0.000000","last_in_change":"2024-08-30T15:58:25.430425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T15:58:50.881 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-08-30T15:58:52.001 INFO:tasks.ceph.osd.4.smithi094.stderr:==00:00:00:26.419 23102== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:52.001 INFO:tasks.ceph.osd.5.smithi094.stderr:==00:00:00:26.431 23100== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:52.740 INFO:tasks.ceph.osd.7.smithi094.stderr:==00:00:00:27.151 23103== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:53.020 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:27.447 23101== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:53.063 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:27.489 22878== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:53.096 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:27.520 22879== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:53.309 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:27.729 22880== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:53.589 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:28.017 22877== Warning: unimplemented fcntl command: 1038 2024-08-30T15:58:57.689 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-30T15:58:58.571 INFO:teuthology.misc.health.smithi050.stdout: 2024-08-30T15:58:58.571 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T15:58:25.430425+0000","last_up_change":"0.000000","last_in_change":"2024-08-30T15:58:25.430425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T15:58:58.583 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-08-30T15:58:59.855 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:58:59.848+0000 5ef1600 -1 osd.4 0 log_to_monitors true 2024-08-30T15:59:00.145 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:00.138+0000 5ef1600 -1 osd.5 0 log_to_monitors true 2024-08-30T15:59:01.181 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:01.174+0000 5ef1600 -1 osd.7 0 log_to_monitors true 2024-08-30T15:59:01.452 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:01.445+0000 5ef1600 -1 osd.6 0 log_to_monitors true 2024-08-30T15:59:01.550 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:01.535+0000 5ef1600 -1 osd.3 0 log_to_monitors true 2024-08-30T15:59:01.837 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:01.830+0000 5ef1600 -1 osd.0 0 log_to_monitors true 2024-08-30T15:59:01.925 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:01.915+0000 5ef1600 -1 osd.2 0 log_to_monitors true 2024-08-30T15:59:02.783 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:02.770+0000 5ef1600 -1 osd.1 0 log_to_monitors true 2024-08-30T15:59:03.314 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:03.308+0000 bdc5640 -1 osd.5 0 waiting for initial osdmap 2024-08-30T15:59:03.700 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:03.686+0000 1038b640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T15:59:05.390 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-30T15:59:05.434 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:05.428+0000 bdc5640 -1 osd.0 0 waiting for initial osdmap 2024-08-30T15:59:05.498 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:05.493+0000 b5ae640 -1 osd.2 0 waiting for initial osdmap 2024-08-30T15:59:05.932 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:05.916+0000 1038b640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T15:59:05.961 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:05.946+0000 1038b640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T15:59:06.032 INFO:teuthology.misc.health.smithi050.stdout: 2024-08-30T15:59:06.033 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":14,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T15:59:05.076841+0000","last_up_change":"2024-08-30T15:59:03.911218+0000","last_in_change":"2024-08-30T15:58:25.430425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":3876025616}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":3876025616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":3876025616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":3876025616}]},"public_addr":"172.21.15.94:6809/3876025616","cluster_addr":"172.21.15.94:6810/3876025616","heartbeat_back_addr":"172.21.15.94:6812/3876025616","heartbeat_front_addr":"172.21.15.94:6811/3876025616","state":["exists","up"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:02.082761+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T15:59:06.048 DEBUG:teuthology.misc:1 of 8 OSDs are up 2024-08-30T15:59:11.693 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:11.530+0000 5ef1600 -1 bdev(0x17c4b8e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-08-30T15:59:11.707 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:11.547+0000 5ef1600 -1 bdev(0x17c4b8e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-08-30T15:59:11.709 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:11.635+0000 5ef1600 -1 bdev(0x17c4b8e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-08-30T15:59:11.709 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:11.663+0000 5ef1600 -1 bdev(0x17c4b8e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-08-30T15:59:11.718 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:11.691+0000 5ef1600 -1 bdev(0x17c4b8e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-08-30T15:59:11.768 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:11.707+0000 5ef1600 -1 bdev(0x17c4b8e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-08-30T15:59:11.881 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:11.717+0000 5ef1600 -1 bdev(0x17c4b8e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-08-30T15:59:11.882 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:11.767+0000 5ef1600 -1 bdev(0x17c4b8e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-08-30T15:59:11.882 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:11.788+0000 5ef1600 -1 bdev(0x17c4b8e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-08-30T15:59:12.060 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:11.880+0000 5ef1600 -1 bdev(0x17c4b8e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-08-30T15:59:12.061 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:11.990+0000 5ef1600 -1 bdev(0x17c4b8e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-08-30T15:59:12.127 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:12.061+0000 5ef1600 -1 bdev(0x17c4b8e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-08-30T15:59:12.856 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-30T15:59:12.861 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:12.858+0000 5ef1600 -1 bdev(0x17c4df70 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-08-30T15:59:12.921 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:12.128+0000 5ef1600 -1 bdev(0x17c4b8e0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-08-30T15:59:13.124 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:12.938+0000 5ef1600 -1 bdev(0x17c4df70 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-08-30T15:59:13.148 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:13.079+0000 5ef1600 -1 bdev(0x15a6df30 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-08-30T15:59:13.237 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:12.983+0000 5ef1600 -1 bdev(0x17c4df70 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-08-30T15:59:13.611 INFO:teuthology.misc.health.smithi050.stdout: 2024-08-30T15:59:13.611 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":19,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T15:59:11.595195+0000","last_up_change":"2024-08-30T15:59:07.233055+0000","last_in_change":"2024-08-30T15:58:25.430425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-30T15:59:08.355540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6812","nonce":1957399494}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6813","nonce":1957399494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6815","nonce":1957399494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6814","nonce":1957399494}]},"public_addr":"172.21.15.50:6812/1957399494","cluster_addr":"172.21.15.50:6813/1957399494","heartbeat_back_addr":"172.21.15.50:6815/1957399494","heartbeat_front_addr":"172.21.15.50:6814/1957399494","state":["exists","up"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6804","nonce":287018800}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6805","nonce":287018800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6807","nonce":287018800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6806","nonce":287018800}]},"public_addr":"172.21.15.50:6804/287018800","cluster_addr":"172.21.15.50:6805/287018800","heartbeat_back_addr":"172.21.15.50:6807/287018800","heartbeat_front_addr":"172.21.15.50:6806/287018800","state":["exists","up"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":3876025616}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":3876025616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":3876025616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":3876025616}]},"public_addr":"172.21.15.94:6809/3876025616","cluster_addr":"172.21.15.94:6810/3876025616","heartbeat_back_addr":"172.21.15.94:6812/3876025616","heartbeat_front_addr":"172.21.15.94:6811/3876025616","state":["exists","up"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.911895+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.197952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:02.082761+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T15:59:13.628 DEBUG:teuthology.misc:3 of 8 OSDs are up 2024-08-30T15:59:13.792 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:13.153+0000 5ef1600 -1 bdev(0x15a6df30 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-08-30T15:59:13.935 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:13.070+0000 5ef1600 -1 bdev(0x17c4df70 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-08-30T15:59:13.936 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:13.122+0000 5ef1600 -1 bdev(0x17c4df70 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-08-30T15:59:13.937 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:13.236+0000 5ef1600 -1 bdev(0x17c4df70 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-08-30T15:59:14.040 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:13.644+0000 5ef1600 -1 bdev(0x15a6df30 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-08-30T15:59:14.084 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:13.791+0000 5ef1600 -1 bdev(0x15a6df30 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-08-30T15:59:14.164 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:14.159+0000 b5ae640 -1 osd.4 0 waiting for initial osdmap 2024-08-30T15:59:14.252 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:13.805+0000 5ef1600 -1 bdev(0x15a6df30 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-08-30T15:59:14.253 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:14.038+0000 5ef1600 -1 bdev(0x15a6df30 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-08-30T15:59:14.610 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:14.172+0000 5ef1600 -1 bdev(0x15ae34c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-08-30T15:59:14.707 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:14.692+0000 14393640 -1 osd.4 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T15:59:15.142 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:14.613+0000 5ef1600 -1 bdev(0x15ae34c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-08-30T15:59:15.174 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:14.707+0000 5ef1600 -1 bdev(0x15ae34c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-08-30T15:59:15.182 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:15.180+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:15.197 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:15.141+0000 5ef1600 -1 bdev(0x15ae34c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 2 2024-08-30T15:59:15.228 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:15.206+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:15.308 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:15.232+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:15.329 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:15.328+0000 5ef1600 -1 bdev(0x15ae34c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 3 2024-08-30T15:59:15.371 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:15.364+0000 bdc5640 -1 osd.6 0 waiting for initial osdmap 2024-08-30T15:59:15.558 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:15.393+0000 5ef1600 -1 bdev(0x15ae34c0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-08-30T15:59:15.563 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:15.556+0000 b5ae640 -1 osd.7 0 waiting for initial osdmap 2024-08-30T15:59:15.594 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:15.357+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:15.594 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:15.576+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:15.596 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:15.595+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:15.863 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:15.849+0000 14393640 -1 osd.6 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T15:59:16.005 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:15.908+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:16.046 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:16.032+0000 14393640 -1 osd.7 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T15:59:16.048 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:16.004+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:16.048 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:16.019+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:16.048 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:16.033+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:16.137 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:16.047+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:16.166 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:16.164+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:16.314 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:16.251+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:16.411 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:16.315+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:16.556 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:16.458+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-08-30T15:59:16.616 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:16.566+0000 5ef1600 -1 bdev(0x17c4b940 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-08-30T15:59:16.781 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:16.773+0000 bdc5640 -1 osd.3 0 waiting for initial osdmap 2024-08-30T15:59:17.251 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:17.232+0000 14393640 -1 osd.3 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T15:59:17.812 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:17.807+0000 bdc5640 -1 osd.1 0 waiting for initial osdmap 2024-08-30T15:59:18.381 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:18.367+0000 14393640 -1 osd.1 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T15:59:20.435 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-30T15:59:21.114 INFO:teuthology.misc.health.smithi050.stdout: 2024-08-30T15:59:21.114 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":23,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T15:59:20.426284+0000","last_up_change":"2024-08-30T15:59:19.280372+0000","last_in_change":"2024-08-30T15:58:25.430425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-30T15:59:08.355540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6812","nonce":1957399494}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6813","nonce":1957399494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6815","nonce":1957399494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6814","nonce":1957399494}]},"public_addr":"172.21.15.50:6812/1957399494","cluster_addr":"172.21.15.50:6813/1957399494","heartbeat_back_addr":"172.21.15.50:6815/1957399494","heartbeat_front_addr":"172.21.15.50:6814/1957399494","state":["exists","up"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6808","nonce":3208178756}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6809","nonce":3208178756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6811","nonce":3208178756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6810","nonce":3208178756}]},"public_addr":"172.21.15.50:6808/3208178756","cluster_addr":"172.21.15.50:6809/3208178756","heartbeat_back_addr":"172.21.15.50:6811/3208178756","heartbeat_front_addr":"172.21.15.50:6810/3208178756","state":["exists","up"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6804","nonce":287018800}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6805","nonce":287018800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6807","nonce":287018800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6806","nonce":287018800}]},"public_addr":"172.21.15.50:6804/287018800","cluster_addr":"172.21.15.50:6805/287018800","heartbeat_back_addr":"172.21.15.50:6807/287018800","heartbeat_front_addr":"172.21.15.50:6806/287018800","state":["exists","up"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6800","nonce":1140717714}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6801","nonce":1140717714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6803","nonce":1140717714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6802","nonce":1140717714}]},"public_addr":"172.21.15.50:6800/1140717714","cluster_addr":"172.21.15.50:6801/1140717714","heartbeat_back_addr":"172.21.15.50:6803/1140717714","heartbeat_front_addr":"172.21.15.50:6802/1140717714","state":["exists","up"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":1468073345}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":1468073345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":1468073345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":1468073345}]},"public_addr":"172.21.15.94:6805/1468073345","cluster_addr":"172.21.15.94:6806/1468073345","heartbeat_back_addr":"172.21.15.94:6808/1468073345","heartbeat_front_addr":"172.21.15.94:6807/1468073345","state":["exists","up"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":3876025616}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":3876025616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":3876025616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":3876025616}]},"public_addr":"172.21.15.94:6809/3876025616","cluster_addr":"172.21.15.94:6810/3876025616","heartbeat_back_addr":"172.21.15.94:6812/3876025616","heartbeat_front_addr":"172.21.15.94:6811/3876025616","state":["exists","up"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":128519122}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":128519122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":128519122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":128519122}]},"public_addr":"172.21.15.94:6801/128519122","cluster_addr":"172.21.15.94:6802/128519122","heartbeat_back_addr":"172.21.15.94:6804/128519122","heartbeat_front_addr":"172.21.15.94:6803/128519122","state":["exists","up"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":68721277}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":68721277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6816","nonce":68721277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":68721277}]},"public_addr":"172.21.15.94:6813/68721277","cluster_addr":"172.21.15.94:6814/68721277","heartbeat_back_addr":"172.21.15.94:6816/68721277","heartbeat_front_addr":"172.21.15.94:6815/68721277","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.911895+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.786543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.197952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.680847+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:01.937600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:02.082761+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.401025+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.129371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T15:59:21.125 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-08-30T15:59:21.125 INFO:tasks.ceph:Creating RBD pool 2024-08-30T15:59:21.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd pool create rbd 8 2024-08-30T15:59:22.727 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'rbd' created 2024-08-30T15:59:22.739 DEBUG:teuthology.orchestra.run.smithi050:> rbd --cluster ceph pool init rbd 2024-08-30T15:59:23.414 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:24.741 22547== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:23.471 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:24.798 22547== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:23.525 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:24.797 22555== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:23.526 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:24.853 22554== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:23.582 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:24.909 22554== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:23.583 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:24.854 22555== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:23.737 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:25.063 22547== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:23.831 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:25.103 22555== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:23.870 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:25.197 22554== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:27.706 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-08-30T15:59:27.707 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-30T15:59:27.707 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-08-30T15:59:27.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-08-30T15:59:27.708 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-30T15:59:28.457 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:59:28.468 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":5,"flags":0,"active_gid":4113,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6800","nonce":1457430448}]},"active_addr":"172.21.15.94:6800/1457430448","active_change":"2024-08-30T15:58:25.607751+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4108,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":2958010077}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":4205708399}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":108091667}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":2157149632}]}]} 2024-08-30T15:59:28.472 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-08-30T15:59:28.472 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-08-30T15:59:28.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-30T15:59:29.211 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:59:29.212 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":28,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T15:59:27.378735+0000","last_up_change":"2024-08-30T15:59:19.280372+0000","last_in_change":"2024-08-30T15:58:25.430425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-30T15:59:08.355540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-30T15:59:21.696796+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6812","nonce":1957399494}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6813","nonce":1957399494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6815","nonce":1957399494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6814","nonce":1957399494}]},"public_addr":"172.21.15.50:6812/1957399494","cluster_addr":"172.21.15.50:6813/1957399494","heartbeat_back_addr":"172.21.15.50:6815/1957399494","heartbeat_front_addr":"172.21.15.50:6814/1957399494","state":["exists","up"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6808","nonce":3208178756}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6809","nonce":3208178756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6811","nonce":3208178756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6810","nonce":3208178756}]},"public_addr":"172.21.15.50:6808/3208178756","cluster_addr":"172.21.15.50:6809/3208178756","heartbeat_back_addr":"172.21.15.50:6811/3208178756","heartbeat_front_addr":"172.21.15.50:6810/3208178756","state":["exists","up"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6804","nonce":287018800}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6805","nonce":287018800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6807","nonce":287018800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6806","nonce":287018800}]},"public_addr":"172.21.15.50:6804/287018800","cluster_addr":"172.21.15.50:6805/287018800","heartbeat_back_addr":"172.21.15.50:6807/287018800","heartbeat_front_addr":"172.21.15.50:6806/287018800","state":["exists","up"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6800","nonce":1140717714}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6801","nonce":1140717714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6803","nonce":1140717714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6802","nonce":1140717714}]},"public_addr":"172.21.15.50:6800/1140717714","cluster_addr":"172.21.15.50:6801/1140717714","heartbeat_back_addr":"172.21.15.50:6803/1140717714","heartbeat_front_addr":"172.21.15.50:6802/1140717714","state":["exists","up"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":1468073345}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":1468073345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":1468073345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":1468073345}]},"public_addr":"172.21.15.94:6805/1468073345","cluster_addr":"172.21.15.94:6806/1468073345","heartbeat_back_addr":"172.21.15.94:6808/1468073345","heartbeat_front_addr":"172.21.15.94:6807/1468073345","state":["exists","up"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":3876025616}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":3876025616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":3876025616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":3876025616}]},"public_addr":"172.21.15.94:6809/3876025616","cluster_addr":"172.21.15.94:6810/3876025616","heartbeat_back_addr":"172.21.15.94:6812/3876025616","heartbeat_front_addr":"172.21.15.94:6811/3876025616","state":["exists","up"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":128519122}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":128519122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":128519122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":128519122}]},"public_addr":"172.21.15.94:6801/128519122","cluster_addr":"172.21.15.94:6802/128519122","heartbeat_back_addr":"172.21.15.94:6804/128519122","heartbeat_front_addr":"172.21.15.94:6803/128519122","state":["exists","up"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":68721277}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":68721277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6816","nonce":68721277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":68721277}]},"public_addr":"172.21.15.94:6813/68721277","cluster_addr":"172.21.15.94:6814/68721277","heartbeat_back_addr":"172.21.15.94:6816/68721277","heartbeat_front_addr":"172.21.15.94:6815/68721277","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.911895+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.786543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.197952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.680847+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:01.937600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:02.082761+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.401025+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.129371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T15:59:29.224 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-08-30T15:59:29.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-30T15:59:29.802 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:59:29.803 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":28,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T15:59:27.378735+0000","last_up_change":"2024-08-30T15:59:19.280372+0000","last_in_change":"2024-08-30T15:58:25.430425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-30T15:59:08.355540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-30T15:59:21.696796+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6812","nonce":1957399494}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6813","nonce":1957399494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6815","nonce":1957399494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6814","nonce":1957399494}]},"public_addr":"172.21.15.50:6812/1957399494","cluster_addr":"172.21.15.50:6813/1957399494","heartbeat_back_addr":"172.21.15.50:6815/1957399494","heartbeat_front_addr":"172.21.15.50:6814/1957399494","state":["exists","up"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6808","nonce":3208178756}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6809","nonce":3208178756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6811","nonce":3208178756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6810","nonce":3208178756}]},"public_addr":"172.21.15.50:6808/3208178756","cluster_addr":"172.21.15.50:6809/3208178756","heartbeat_back_addr":"172.21.15.50:6811/3208178756","heartbeat_front_addr":"172.21.15.50:6810/3208178756","state":["exists","up"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6804","nonce":287018800}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6805","nonce":287018800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6807","nonce":287018800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6806","nonce":287018800}]},"public_addr":"172.21.15.50:6804/287018800","cluster_addr":"172.21.15.50:6805/287018800","heartbeat_back_addr":"172.21.15.50:6807/287018800","heartbeat_front_addr":"172.21.15.50:6806/287018800","state":["exists","up"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6800","nonce":1140717714}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6801","nonce":1140717714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6803","nonce":1140717714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6802","nonce":1140717714}]},"public_addr":"172.21.15.50:6800/1140717714","cluster_addr":"172.21.15.50:6801/1140717714","heartbeat_back_addr":"172.21.15.50:6803/1140717714","heartbeat_front_addr":"172.21.15.50:6802/1140717714","state":["exists","up"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":1468073345}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":1468073345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":1468073345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":1468073345}]},"public_addr":"172.21.15.94:6805/1468073345","cluster_addr":"172.21.15.94:6806/1468073345","heartbeat_back_addr":"172.21.15.94:6808/1468073345","heartbeat_front_addr":"172.21.15.94:6807/1468073345","state":["exists","up"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":3876025616}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":3876025616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":3876025616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":3876025616}]},"public_addr":"172.21.15.94:6809/3876025616","cluster_addr":"172.21.15.94:6810/3876025616","heartbeat_back_addr":"172.21.15.94:6812/3876025616","heartbeat_front_addr":"172.21.15.94:6811/3876025616","state":["exists","up"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":128519122}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":128519122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":128519122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":128519122}]},"public_addr":"172.21.15.94:6801/128519122","cluster_addr":"172.21.15.94:6802/128519122","heartbeat_back_addr":"172.21.15.94:6804/128519122","heartbeat_front_addr":"172.21.15.94:6803/128519122","state":["exists","up"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":68721277}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":68721277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6816","nonce":68721277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":68721277}]},"public_addr":"172.21.15.94:6813/68721277","cluster_addr":"172.21.15.94:6814/68721277","heartbeat_back_addr":"172.21.15.94:6816/68721277","heartbeat_front_addr":"172.21.15.94:6815/68721277","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.911895+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.786543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.197952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.680847+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:01.937600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:02.082761+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.401025+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.129371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T15:59:29.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-08-30T15:59:29.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-08-30T15:59:29.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-08-30T15:59:29.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-08-30T15:59:29.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-08-30T15:59:29.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-08-30T15:59:29.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-08-30T15:59:29.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-08-30T15:59:30.229 INFO:teuthology.orchestra.run.smithi050.stdout:68719476743 2024-08-30T15:59:30.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-30T15:59:30.235 INFO:teuthology.orchestra.run.smithi050.stdout:90194313226 2024-08-30T15:59:30.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-08-30T15:59:30.257 INFO:teuthology.orchestra.run.smithi050.stdout:94489280523 2024-08-30T15:59:30.257 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-30T15:59:30.275 INFO:teuthology.orchestra.run.smithi050.stdout:90194313225 2024-08-30T15:59:30.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-08-30T15:59:30.298 INFO:teuthology.orchestra.run.smithi050.stdout:55834574855 2024-08-30T15:59:30.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-30T15:59:30.307 INFO:teuthology.orchestra.run.smithi050.stdout:90194313225 2024-08-30T15:59:30.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-30T15:59:30.334 INFO:teuthology.orchestra.run.smithi050.stdout:64424509446 2024-08-30T15:59:30.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-30T15:59:30.353 INFO:teuthology.orchestra.run.smithi050.stdout:85899345929 2024-08-30T15:59:30.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-30T15:59:30.983 INFO:teuthology.orchestra.run.smithi050.stdout:68719476742 2024-08-30T15:59:30.993 INFO:teuthology.orchestra.run.smithi050.stdout:94489280521 2024-08-30T15:59:30.997 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476742 for osd.2 2024-08-30T15:59:31.011 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280523 got 94489280521 for osd.1 2024-08-30T15:59:31.162 INFO:teuthology.orchestra.run.smithi050.stdout:90194313223 2024-08-30T15:59:31.174 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313223 for osd.6 2024-08-30T15:59:31.412 INFO:teuthology.orchestra.run.smithi050.stdout:90194313224 2024-08-30T15:59:31.430 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313226 got 90194313224 for osd.7 2024-08-30T15:59:31.844 INFO:teuthology.orchestra.run.smithi050.stdout:64424509445 2024-08-30T15:59:31.857 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.0 2024-08-30T15:59:31.896 INFO:teuthology.orchestra.run.smithi050.stdout:55834574854 2024-08-30T15:59:31.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2024-08-30T15:59:31.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-30T15:59:32.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-30T15:59:32.069 INFO:teuthology.orchestra.run.smithi050.stdout:85899345928 2024-08-30T15:59:32.087 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345928 for osd.4 2024-08-30T15:59:32.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-08-30T15:59:32.377 INFO:teuthology.orchestra.run.smithi050.stdout:90194313224 2024-08-30T15:59:32.390 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313224 for osd.3 2024-08-30T15:59:32.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-08-30T15:59:32.562 INFO:teuthology.orchestra.run.smithi050.stdout:68719476742 2024-08-30T15:59:32.579 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476742 for osd.2 2024-08-30T15:59:32.799 INFO:teuthology.orchestra.run.smithi050.stdout:90194313224 2024-08-30T15:59:32.817 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313224 for osd.6 2024-08-30T15:59:32.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-30T15:59:32.864 INFO:teuthology.orchestra.run.smithi050.stdout:94489280521 2024-08-30T15:59:32.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280523 got 94489280521 for osd.1 2024-08-30T15:59:32.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-30T15:59:33.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-30T15:59:33.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-30T15:59:33.416 INFO:teuthology.orchestra.run.smithi050.stdout:90194313226 2024-08-30T15:59:33.434 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313226 got 90194313226 for osd.7 2024-08-30T15:59:33.434 DEBUG:teuthology.parallel:result is None 2024-08-30T15:59:33.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-30T15:59:33.728 INFO:teuthology.orchestra.run.smithi050.stdout:55834574855 2024-08-30T15:59:33.741 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2024-08-30T15:59:33.741 DEBUG:teuthology.parallel:result is None 2024-08-30T15:59:33.776 INFO:teuthology.orchestra.run.smithi050.stdout:85899345929 2024-08-30T15:59:33.790 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345929 for osd.4 2024-08-30T15:59:33.790 DEBUG:teuthology.parallel:result is None 2024-08-30T15:59:33.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-08-30T15:59:33.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-30T15:59:33.935 INFO:teuthology.orchestra.run.smithi050.stdout:64424509447 2024-08-30T15:59:33.950 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.0 2024-08-30T15:59:33.950 DEBUG:teuthology.parallel:result is None 2024-08-30T15:59:33.993 INFO:teuthology.orchestra.run.smithi050.stdout:90194313225 2024-08-30T15:59:34.008 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313225 for osd.3 2024-08-30T15:59:34.009 DEBUG:teuthology.parallel:result is None 2024-08-30T15:59:34.202 INFO:teuthology.orchestra.run.smithi050.stdout:68719476743 2024-08-30T15:59:34.218 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476743 for osd.2 2024-08-30T15:59:34.219 DEBUG:teuthology.parallel:result is None 2024-08-30T15:59:34.829 INFO:teuthology.orchestra.run.smithi050.stdout:90194313225 2024-08-30T15:59:34.842 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313225 for osd.6 2024-08-30T15:59:34.842 DEBUG:teuthology.parallel:result is None 2024-08-30T15:59:34.889 INFO:teuthology.orchestra.run.smithi050.stdout:94489280523 2024-08-30T15:59:34.902 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280523 got 94489280523 for osd.1 2024-08-30T15:59:34.902 DEBUG:teuthology.parallel:result is None 2024-08-30T15:59:34.902 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-30T15:59:34.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T15:59:35.457 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:59:35.458 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T15:59:35.469 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-08-30T15:59:34.359227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218152,"kb_used_data":1964,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756568,"statfs":{"total":773094113280,"available":772870725632,"internally_reserved":0,"allocated":2011136,"data_stored":1473734,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.914442"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.643715+0000","last_change":"2024-08-30T15:59:27.646756+0000","last_active":"2024-08-30T15:59:27.643715+0000","last_peered":"2024-08-30T15:59:27.643715+0000","last_clean":"2024-08-30T15:59:27.643715+0000","last_became_active":"2024-08-30T15:59:25.342321+0000","last_became_peered":"2024-08-30T15:59:25.342321+0000","last_unstale":"2024-08-30T15:59:27.643715+0000","last_undegraded":"2024-08-30T15:59:27.643715+0000","last_fullsized":"2024-08-30T15:59:27.643715+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:53:42.927368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.116914037,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:28.120693+0000","last_change":"2024-08-30T15:59:28.156737+0000","last_active":"2024-08-30T15:59:28.120693+0000","last_peered":"2024-08-30T15:59:28.120693+0000","last_clean":"2024-08-30T15:59:28.120693+0000","last_became_active":"2024-08-30T15:59:25.662524+0000","last_became_peered":"2024-08-30T15:59:25.662524+0000","last_unstale":"2024-08-30T15:59:28.120693+0000","last_undegraded":"2024-08-30T15:59:28.120693+0000","last_fullsized":"2024-08-30T15:59:28.120693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:05:30.092449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064329163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.629302+0000","last_change":"2024-08-30T15:59:27.633478+0000","last_active":"2024-08-30T15:59:27.629302+0000","last_peered":"2024-08-30T15:59:27.629302+0000","last_clean":"2024-08-30T15:59:27.629302+0000","last_became_active":"2024-08-30T15:59:24.572730+0000","last_became_peered":"2024-08-30T15:59:24.572730+0000","last_unstale":"2024-08-30T15:59:27.629302+0000","last_undegraded":"2024-08-30T15:59:27.629302+0000","last_fullsized":"2024-08-30T15:59:27.629302+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:36:20.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-08-30T15:59:34.323340+0000","last_change":"2024-08-30T15:59:27.584250+0000","last_active":"2024-08-30T15:59:34.323340+0000","last_peered":"2024-08-30T15:59:34.323340+0000","last_clean":"2024-08-30T15:59:34.323340+0000","last_became_active":"2024-08-30T15:59:24.687233+0000","last_became_peered":"2024-08-30T15:59:24.687233+0000","last_unstale":"2024-08-30T15:59:34.323340+0000","last_undegraded":"2024-08-30T15:59:34.323340+0000","last_fullsized":"2024-08-30T15:59:34.323340+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:49:09.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.623039+0000","last_change":"2024-08-30T15:59:27.623039+0000","last_active":"2024-08-30T15:59:27.623039+0000","last_peered":"2024-08-30T15:59:27.623039+0000","last_clean":"2024-08-30T15:59:27.623039+0000","last_became_active":"2024-08-30T15:59:24.688486+0000","last_became_peered":"2024-08-30T15:59:24.688486+0000","last_unstale":"2024-08-30T15:59:27.623039+0000","last_undegraded":"2024-08-30T15:59:27.623039+0000","last_fullsized":"2024-08-30T15:59:27.623039+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:22:51.391036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.783650+0000","last_change":"2024-08-30T15:59:27.825532+0000","last_active":"2024-08-30T15:59:27.783650+0000","last_peered":"2024-08-30T15:59:27.783650+0000","last_clean":"2024-08-30T15:59:27.783650+0000","last_became_active":"2024-08-30T15:59:25.469390+0000","last_became_peered":"2024-08-30T15:59:25.469390+0000","last_unstale":"2024-08-30T15:59:27.783650+0000","last_undegraded":"2024-08-30T15:59:27.783650+0000","last_fullsized":"2024-08-30T15:59:27.783650+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T17:46:39.688911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.542867+0000","last_change":"2024-08-30T15:59:27.583986+0000","last_active":"2024-08-30T15:59:27.542867+0000","last_peered":"2024-08-30T15:59:27.542867+0000","last_clean":"2024-08-30T15:59:27.542867+0000","last_became_active":"2024-08-30T15:59:24.528536+0000","last_became_peered":"2024-08-30T15:59:24.528536+0000","last_unstale":"2024-08-30T15:59:27.542867+0000","last_undegraded":"2024-08-30T15:59:27.542867+0000","last_fullsized":"2024-08-30T15:59:27.542867+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:18:40.602710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"25'1","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.630431+0000","last_change":"2024-08-30T15:59:27.636613+0000","last_active":"2024-08-30T15:59:27.630431+0000","last_peered":"2024-08-30T15:59:27.630431+0000","last_clean":"2024-08-30T15:59:27.630431+0000","last_became_active":"2024-08-30T15:59:25.021268+0000","last_became_peered":"2024-08-30T15:59:25.021268+0000","last_unstale":"2024-08-30T15:59:27.630431+0000","last_undegraded":"2024-08-30T15:59:27.630431+0000","last_fullsized":"2024-08-30T15:59:27.630431+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T00:56:28.767086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113256018,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"21'76","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-30T15:59:26.500893+0000","last_change":"2024-08-30T15:59:21.623783+0000","last_active":"2024-08-30T15:59:26.500893+0000","last_peered":"2024-08-30T15:59:26.500893+0000","last_clean":"2024-08-30T15:59:26.500893+0000","last_became_active":"2024-08-30T15:59:20.848924+0000","last_became_peered":"2024-08-30T15:59:20.848924+0000","last_unstale":"2024-08-30T15:59:26.500893+0000","last_undegraded":"2024-08-30T15:59:26.500893+0000","last_fullsized":"2024-08-30T15:59:26.500893+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:14:52.575421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":795680,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280523,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":482592,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.182}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.6020000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.196999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.433999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.541}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.257}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.047999999999998}]}]},{"osd":7,"up_from":21,"seq":90194313227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":84752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.343999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0370000000000008}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8420000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3409999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.868}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.494}]}]},{"osd":6,"up_from":21,"seq":90194313226,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4260000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.971}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2569999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.574}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.885}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.448}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.364000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313226,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2670000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.108000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.2170000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8439999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.926}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.467000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.714}]}]},{"osd":4,"up_from":20,"seq":85899345930,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":84752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.853}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2759999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.316000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.032999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.603999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.625}]}]},{"osd":2,"up_from":16,"seq":68719476744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":84752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.523}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7200000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.295}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.815}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.694000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.82}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.731999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":84752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.789999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.341000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.498000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.734999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.542999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":28.379999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.353999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":482592,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.381}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1379999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.327999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.548999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.057}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7949999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.5}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T15:59:35.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T15:59:36.010 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:59:36.011 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T15:59:36.022 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2024-08-30T15:59:35.872060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218152,"kb_used_data":1964,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756568,"statfs":{"total":773094113280,"available":772870725632,"internally_reserved":0,"allocated":2011136,"data_stored":1473734,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.443358"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.643715+0000","last_change":"2024-08-30T15:59:27.646756+0000","last_active":"2024-08-30T15:59:27.643715+0000","last_peered":"2024-08-30T15:59:27.643715+0000","last_clean":"2024-08-30T15:59:27.643715+0000","last_became_active":"2024-08-30T15:59:25.342321+0000","last_became_peered":"2024-08-30T15:59:25.342321+0000","last_unstale":"2024-08-30T15:59:27.643715+0000","last_undegraded":"2024-08-30T15:59:27.643715+0000","last_fullsized":"2024-08-30T15:59:27.643715+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:53:42.927368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.116914037,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:28.120693+0000","last_change":"2024-08-30T15:59:28.156737+0000","last_active":"2024-08-30T15:59:28.120693+0000","last_peered":"2024-08-30T15:59:28.120693+0000","last_clean":"2024-08-30T15:59:28.120693+0000","last_became_active":"2024-08-30T15:59:25.662524+0000","last_became_peered":"2024-08-30T15:59:25.662524+0000","last_unstale":"2024-08-30T15:59:28.120693+0000","last_undegraded":"2024-08-30T15:59:28.120693+0000","last_fullsized":"2024-08-30T15:59:28.120693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:05:30.092449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064329163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.629302+0000","last_change":"2024-08-30T15:59:27.633478+0000","last_active":"2024-08-30T15:59:27.629302+0000","last_peered":"2024-08-30T15:59:27.629302+0000","last_clean":"2024-08-30T15:59:27.629302+0000","last_became_active":"2024-08-30T15:59:24.572730+0000","last_became_peered":"2024-08-30T15:59:24.572730+0000","last_unstale":"2024-08-30T15:59:27.629302+0000","last_undegraded":"2024-08-30T15:59:27.629302+0000","last_fullsized":"2024-08-30T15:59:27.629302+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:36:20.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-08-30T15:59:34.323340+0000","last_change":"2024-08-30T15:59:27.584250+0000","last_active":"2024-08-30T15:59:34.323340+0000","last_peered":"2024-08-30T15:59:34.323340+0000","last_clean":"2024-08-30T15:59:34.323340+0000","last_became_active":"2024-08-30T15:59:24.687233+0000","last_became_peered":"2024-08-30T15:59:24.687233+0000","last_unstale":"2024-08-30T15:59:34.323340+0000","last_undegraded":"2024-08-30T15:59:34.323340+0000","last_fullsized":"2024-08-30T15:59:34.323340+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:49:09.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.623039+0000","last_change":"2024-08-30T15:59:27.623039+0000","last_active":"2024-08-30T15:59:27.623039+0000","last_peered":"2024-08-30T15:59:27.623039+0000","last_clean":"2024-08-30T15:59:27.623039+0000","last_became_active":"2024-08-30T15:59:24.688486+0000","last_became_peered":"2024-08-30T15:59:24.688486+0000","last_unstale":"2024-08-30T15:59:27.623039+0000","last_undegraded":"2024-08-30T15:59:27.623039+0000","last_fullsized":"2024-08-30T15:59:27.623039+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:22:51.391036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.783650+0000","last_change":"2024-08-30T15:59:27.825532+0000","last_active":"2024-08-30T15:59:27.783650+0000","last_peered":"2024-08-30T15:59:27.783650+0000","last_clean":"2024-08-30T15:59:27.783650+0000","last_became_active":"2024-08-30T15:59:25.469390+0000","last_became_peered":"2024-08-30T15:59:25.469390+0000","last_unstale":"2024-08-30T15:59:27.783650+0000","last_undegraded":"2024-08-30T15:59:27.783650+0000","last_fullsized":"2024-08-30T15:59:27.783650+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T17:46:39.688911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.542867+0000","last_change":"2024-08-30T15:59:27.583986+0000","last_active":"2024-08-30T15:59:27.542867+0000","last_peered":"2024-08-30T15:59:27.542867+0000","last_clean":"2024-08-30T15:59:27.542867+0000","last_became_active":"2024-08-30T15:59:24.528536+0000","last_became_peered":"2024-08-30T15:59:24.528536+0000","last_unstale":"2024-08-30T15:59:27.542867+0000","last_undegraded":"2024-08-30T15:59:27.542867+0000","last_fullsized":"2024-08-30T15:59:27.542867+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:18:40.602710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"25'1","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.630431+0000","last_change":"2024-08-30T15:59:27.636613+0000","last_active":"2024-08-30T15:59:27.630431+0000","last_peered":"2024-08-30T15:59:27.630431+0000","last_clean":"2024-08-30T15:59:27.630431+0000","last_became_active":"2024-08-30T15:59:25.021268+0000","last_became_peered":"2024-08-30T15:59:25.021268+0000","last_unstale":"2024-08-30T15:59:27.630431+0000","last_undegraded":"2024-08-30T15:59:27.630431+0000","last_fullsized":"2024-08-30T15:59:27.630431+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T00:56:28.767086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113256018,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"21'76","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.455967+0000","last_change":"2024-08-30T15:59:21.623783+0000","last_active":"2024-08-30T15:59:27.455967+0000","last_peered":"2024-08-30T15:59:27.455967+0000","last_clean":"2024-08-30T15:59:27.455967+0000","last_became_active":"2024-08-30T15:59:20.848924+0000","last_became_peered":"2024-08-30T15:59:20.848924+0000","last_unstale":"2024-08-30T15:59:27.455967+0000","last_undegraded":"2024-08-30T15:59:27.455967+0000","last_fullsized":"2024-08-30T15:59:27.455967+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:14:52.575421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":795680,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280524,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":482592,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.731}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.25}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.129000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.19}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.866}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.814}]}]},{"osd":7,"up_from":21,"seq":90194313227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":84752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.343999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0370000000000008}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8420000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3409999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.868}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.494}]}]},{"osd":6,"up_from":21,"seq":90194313226,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4260000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.971}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2569999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.574}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.885}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.448}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.364000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313226,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2670000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.108000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.2170000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8439999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.926}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.467000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.714}]}]},{"osd":4,"up_from":20,"seq":85899345930,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":84752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.853}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2759999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.316000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.648999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.032999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.603999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.625}]}]},{"osd":2,"up_from":16,"seq":68719476744,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":84752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.523}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7200000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.295}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.815}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.694000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.82}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.731999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":84752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.789999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.341000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.498000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.734999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.542999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":28.379999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.353999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":482592,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.381}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1379999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.327999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.548999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.057}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7949999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.5}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T15:59:36.023 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-30T15:59:36.023 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-30T15:59:36.023 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-08-30T15:59:36.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-08-30T15:59:36.606 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:59:36.607 INFO:teuthology.orchestra.run.smithi050.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-30T15:59:36.619 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-08-30T15:59:36.619 INFO:teuthology.run_tasks:Running task thrashosds... 2024-08-30T15:59:36.630 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-08-30T15:59:36.630 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-ksirivad-backport-reef-57906'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80, 'osd op thread timeout': 150}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW OPS\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': 'e5753ca846ae583f991bb12d29862578d4f67590', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'e5753ca846ae583f991bb12d29862578d4f67590'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-ksirivad-backport-reef-57906', 'sha1': 'e5753ca846ae583f991bb12d29862578d4f67590'}} 2024-08-30T15:59:36.630 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-08-30T15:59:36.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-08-30T15:59:37.369 INFO:tasks.thrashosds:Beginning thrashosds... 2024-08-30T15:59:37.370 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-30T15:59:37.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T15:59:37.908 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:59:37.908 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T15:59:37.920 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2024-08-30T15:59:37.872381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218172,"kb_used_data":1988,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756548,"statfs":{"total":773094113280,"available":772870705152,"internally_reserved":0,"allocated":2035712,"data_stored":1486772,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001623"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.643715+0000","last_change":"2024-08-30T15:59:27.646756+0000","last_active":"2024-08-30T15:59:27.643715+0000","last_peered":"2024-08-30T15:59:27.643715+0000","last_clean":"2024-08-30T15:59:27.643715+0000","last_became_active":"2024-08-30T15:59:25.342321+0000","last_became_peered":"2024-08-30T15:59:25.342321+0000","last_unstale":"2024-08-30T15:59:27.643715+0000","last_undegraded":"2024-08-30T15:59:27.643715+0000","last_fullsized":"2024-08-30T15:59:27.643715+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:53:42.927368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.116914037,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-08-30T15:59:34.401506+0000","last_change":"2024-08-30T15:59:28.156737+0000","last_active":"2024-08-30T15:59:34.401506+0000","last_peered":"2024-08-30T15:59:34.401506+0000","last_clean":"2024-08-30T15:59:34.401506+0000","last_became_active":"2024-08-30T15:59:25.662524+0000","last_became_peered":"2024-08-30T15:59:25.662524+0000","last_unstale":"2024-08-30T15:59:34.401506+0000","last_undegraded":"2024-08-30T15:59:34.401506+0000","last_fullsized":"2024-08-30T15:59:34.401506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:05:30.092449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064329163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.629302+0000","last_change":"2024-08-30T15:59:27.633478+0000","last_active":"2024-08-30T15:59:27.629302+0000","last_peered":"2024-08-30T15:59:27.629302+0000","last_clean":"2024-08-30T15:59:27.629302+0000","last_became_active":"2024-08-30T15:59:24.572730+0000","last_became_peered":"2024-08-30T15:59:24.572730+0000","last_unstale":"2024-08-30T15:59:27.629302+0000","last_undegraded":"2024-08-30T15:59:27.629302+0000","last_fullsized":"2024-08-30T15:59:27.629302+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:36:20.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-08-30T15:59:34.323340+0000","last_change":"2024-08-30T15:59:27.584250+0000","last_active":"2024-08-30T15:59:34.323340+0000","last_peered":"2024-08-30T15:59:34.323340+0000","last_clean":"2024-08-30T15:59:34.323340+0000","last_became_active":"2024-08-30T15:59:24.687233+0000","last_became_peered":"2024-08-30T15:59:24.687233+0000","last_unstale":"2024-08-30T15:59:34.323340+0000","last_undegraded":"2024-08-30T15:59:34.323340+0000","last_fullsized":"2024-08-30T15:59:34.323340+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:49:09.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.623039+0000","last_change":"2024-08-30T15:59:27.623039+0000","last_active":"2024-08-30T15:59:27.623039+0000","last_peered":"2024-08-30T15:59:27.623039+0000","last_clean":"2024-08-30T15:59:27.623039+0000","last_became_active":"2024-08-30T15:59:24.688486+0000","last_became_peered":"2024-08-30T15:59:24.688486+0000","last_unstale":"2024-08-30T15:59:27.623039+0000","last_undegraded":"2024-08-30T15:59:27.623039+0000","last_fullsized":"2024-08-30T15:59:27.623039+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:22:51.391036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.783650+0000","last_change":"2024-08-30T15:59:27.825532+0000","last_active":"2024-08-30T15:59:27.783650+0000","last_peered":"2024-08-30T15:59:27.783650+0000","last_clean":"2024-08-30T15:59:27.783650+0000","last_became_active":"2024-08-30T15:59:25.469390+0000","last_became_peered":"2024-08-30T15:59:25.469390+0000","last_unstale":"2024-08-30T15:59:27.783650+0000","last_undegraded":"2024-08-30T15:59:27.783650+0000","last_fullsized":"2024-08-30T15:59:27.783650+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T17:46:39.688911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.542867+0000","last_change":"2024-08-30T15:59:27.583986+0000","last_active":"2024-08-30T15:59:27.542867+0000","last_peered":"2024-08-30T15:59:27.542867+0000","last_clean":"2024-08-30T15:59:27.542867+0000","last_became_active":"2024-08-30T15:59:24.528536+0000","last_became_peered":"2024-08-30T15:59:24.528536+0000","last_unstale":"2024-08-30T15:59:27.542867+0000","last_undegraded":"2024-08-30T15:59:27.542867+0000","last_fullsized":"2024-08-30T15:59:27.542867+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:18:40.602710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"25'1","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.630431+0000","last_change":"2024-08-30T15:59:27.636613+0000","last_active":"2024-08-30T15:59:27.630431+0000","last_peered":"2024-08-30T15:59:27.630431+0000","last_clean":"2024-08-30T15:59:27.630431+0000","last_became_active":"2024-08-30T15:59:25.021268+0000","last_became_peered":"2024-08-30T15:59:25.021268+0000","last_unstale":"2024-08-30T15:59:27.630431+0000","last_undegraded":"2024-08-30T15:59:27.630431+0000","last_fullsized":"2024-08-30T15:59:27.630431+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T00:56:28.767086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113256018,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"21'76","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.455967+0000","last_change":"2024-08-30T15:59:21.623783+0000","last_active":"2024-08-30T15:59:27.455967+0000","last_peered":"2024-08-30T15:59:27.455967+0000","last_clean":"2024-08-30T15:59:27.455967+0000","last_became_active":"2024-08-30T15:59:20.848924+0000","last_became_peered":"2024-08-30T15:59:20.848924+0000","last_unstale":"2024-08-30T15:59:27.455967+0000","last_undegraded":"2024-08-30T15:59:27.455967+0000","last_fullsized":"2024-08-30T15:59:27.455967+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:14:52.575421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":795680,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280524,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":482592,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.731}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.25}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.129000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.19}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.866}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.814}]}]},{"osd":7,"up_from":21,"seq":90194313227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":84752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.343999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0370000000000008}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8420000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3409999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.868}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.494}]}]},{"osd":6,"up_from":21,"seq":90194313226,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4260000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.971}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2569999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.574}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.885}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.448}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.364000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313226,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2670000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.108000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.2170000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8439999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.926}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.467000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.714}]}]},{"osd":4,"up_from":20,"seq":85899345931,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":89098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.2229999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6749999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.802}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.407}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.625}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.161}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.815}]}]},{"osd":2,"up_from":16,"seq":68719476745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":89098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.500999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.311999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.584}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.380000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.574999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.236000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.038}]}]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":89098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.202999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9379999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.331}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.965}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.865}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1229999999999993}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.263999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":482592,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.381}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1379999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.327999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.548999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.057}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7949999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.5}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T15:59:37.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T15:59:38.469 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:59:38.470 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T15:59:38.486 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2024-08-30T15:59:37.872381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218172,"kb_used_data":1988,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756548,"statfs":{"total":773094113280,"available":772870705152,"internally_reserved":0,"allocated":2035712,"data_stored":1486772,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001623"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.643715+0000","last_change":"2024-08-30T15:59:27.646756+0000","last_active":"2024-08-30T15:59:27.643715+0000","last_peered":"2024-08-30T15:59:27.643715+0000","last_clean":"2024-08-30T15:59:27.643715+0000","last_became_active":"2024-08-30T15:59:25.342321+0000","last_became_peered":"2024-08-30T15:59:25.342321+0000","last_unstale":"2024-08-30T15:59:27.643715+0000","last_undegraded":"2024-08-30T15:59:27.643715+0000","last_fullsized":"2024-08-30T15:59:27.643715+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:53:42.927368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.116914037,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-08-30T15:59:34.401506+0000","last_change":"2024-08-30T15:59:28.156737+0000","last_active":"2024-08-30T15:59:34.401506+0000","last_peered":"2024-08-30T15:59:34.401506+0000","last_clean":"2024-08-30T15:59:34.401506+0000","last_became_active":"2024-08-30T15:59:25.662524+0000","last_became_peered":"2024-08-30T15:59:25.662524+0000","last_unstale":"2024-08-30T15:59:34.401506+0000","last_undegraded":"2024-08-30T15:59:34.401506+0000","last_fullsized":"2024-08-30T15:59:34.401506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:05:30.092449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064329163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.629302+0000","last_change":"2024-08-30T15:59:27.633478+0000","last_active":"2024-08-30T15:59:27.629302+0000","last_peered":"2024-08-30T15:59:27.629302+0000","last_clean":"2024-08-30T15:59:27.629302+0000","last_became_active":"2024-08-30T15:59:24.572730+0000","last_became_peered":"2024-08-30T15:59:24.572730+0000","last_unstale":"2024-08-30T15:59:27.629302+0000","last_undegraded":"2024-08-30T15:59:27.629302+0000","last_fullsized":"2024-08-30T15:59:27.629302+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:36:20.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-08-30T15:59:34.323340+0000","last_change":"2024-08-30T15:59:27.584250+0000","last_active":"2024-08-30T15:59:34.323340+0000","last_peered":"2024-08-30T15:59:34.323340+0000","last_clean":"2024-08-30T15:59:34.323340+0000","last_became_active":"2024-08-30T15:59:24.687233+0000","last_became_peered":"2024-08-30T15:59:24.687233+0000","last_unstale":"2024-08-30T15:59:34.323340+0000","last_undegraded":"2024-08-30T15:59:34.323340+0000","last_fullsized":"2024-08-30T15:59:34.323340+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:49:09.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.623039+0000","last_change":"2024-08-30T15:59:27.623039+0000","last_active":"2024-08-30T15:59:27.623039+0000","last_peered":"2024-08-30T15:59:27.623039+0000","last_clean":"2024-08-30T15:59:27.623039+0000","last_became_active":"2024-08-30T15:59:24.688486+0000","last_became_peered":"2024-08-30T15:59:24.688486+0000","last_unstale":"2024-08-30T15:59:27.623039+0000","last_undegraded":"2024-08-30T15:59:27.623039+0000","last_fullsized":"2024-08-30T15:59:27.623039+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:22:51.391036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.783650+0000","last_change":"2024-08-30T15:59:27.825532+0000","last_active":"2024-08-30T15:59:27.783650+0000","last_peered":"2024-08-30T15:59:27.783650+0000","last_clean":"2024-08-30T15:59:27.783650+0000","last_became_active":"2024-08-30T15:59:25.469390+0000","last_became_peered":"2024-08-30T15:59:25.469390+0000","last_unstale":"2024-08-30T15:59:27.783650+0000","last_undegraded":"2024-08-30T15:59:27.783650+0000","last_fullsized":"2024-08-30T15:59:27.783650+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T17:46:39.688911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.542867+0000","last_change":"2024-08-30T15:59:27.583986+0000","last_active":"2024-08-30T15:59:27.542867+0000","last_peered":"2024-08-30T15:59:27.542867+0000","last_clean":"2024-08-30T15:59:27.542867+0000","last_became_active":"2024-08-30T15:59:24.528536+0000","last_became_peered":"2024-08-30T15:59:24.528536+0000","last_unstale":"2024-08-30T15:59:27.542867+0000","last_undegraded":"2024-08-30T15:59:27.542867+0000","last_fullsized":"2024-08-30T15:59:27.542867+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:18:40.602710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"25'1","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.630431+0000","last_change":"2024-08-30T15:59:27.636613+0000","last_active":"2024-08-30T15:59:27.630431+0000","last_peered":"2024-08-30T15:59:27.630431+0000","last_clean":"2024-08-30T15:59:27.630431+0000","last_became_active":"2024-08-30T15:59:25.021268+0000","last_became_peered":"2024-08-30T15:59:25.021268+0000","last_unstale":"2024-08-30T15:59:27.630431+0000","last_undegraded":"2024-08-30T15:59:27.630431+0000","last_fullsized":"2024-08-30T15:59:27.630431+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T00:56:28.767086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113256018,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"21'76","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-30T15:59:27.455967+0000","last_change":"2024-08-30T15:59:21.623783+0000","last_active":"2024-08-30T15:59:27.455967+0000","last_peered":"2024-08-30T15:59:27.455967+0000","last_clean":"2024-08-30T15:59:27.455967+0000","last_became_active":"2024-08-30T15:59:20.848924+0000","last_became_peered":"2024-08-30T15:59:20.848924+0000","last_unstale":"2024-08-30T15:59:27.455967+0000","last_undegraded":"2024-08-30T15:59:27.455967+0000","last_fullsized":"2024-08-30T15:59:27.455967+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:14:52.575421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":795680,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280524,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":482592,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.731}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.25}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.129000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.19}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.866}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.814}]}]},{"osd":7,"up_from":21,"seq":90194313227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":84752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.343999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0370000000000008}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8420000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3409999999999993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.868}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.494}]}]},{"osd":6,"up_from":21,"seq":90194313226,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4260000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.971}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2569999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.574}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.885}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.448}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.364000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313226,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2670000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.108000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.2170000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8439999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.926}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.467000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.714}]}]},{"osd":4,"up_from":20,"seq":85899345931,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":89098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.2229999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6749999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.802}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.407}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.625}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.161}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.815}]}]},{"osd":2,"up_from":16,"seq":68719476745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":89098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.500999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.311999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.584}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.380000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.574999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.236000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.038}]}]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":89098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.202999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9379999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.331}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.965}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.865}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1229999999999993}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.263999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":482592,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.381}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1379999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.327999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.548999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.057}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7949999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.5}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T15:59:38.487 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-30T15:59:38.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-08-30T15:59:39.144 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 29 2024-08-30T15:59:39.145 INFO:teuthology.orchestra.run.smithi050.stdout:fsid 9d72e2e2-0554-42af-8e62-9b1a3a804112 2024-08-30T15:59:39.145 INFO:teuthology.orchestra.run.smithi050.stdout:created 2024-08-30T15:58:20.129410+0000 2024-08-30T15:59:39.145 INFO:teuthology.orchestra.run.smithi050.stdout:modified 2024-08-30T15:59:33.399292+0000 2024-08-30T15:59:39.145 INFO:teuthology.orchestra.run.smithi050.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-08-30T15:59:39.145 INFO:teuthology.orchestra.run.smithi050.stdout:crush_version 7 2024-08-30T15:59:39.145 INFO:teuthology.orchestra.run.smithi050.stdout:full_ratio 0.95 2024-08-30T15:59:39.145 INFO:teuthology.orchestra.run.smithi050.stdout:backfillfull_ratio 0.9 2024-08-30T15:59:39.145 INFO:teuthology.orchestra.run.smithi050.stdout:nearfull_ratio 0.85 2024-08-30T15:59:39.145 INFO:teuthology.orchestra.run.smithi050.stdout:require_min_compat_client luminous 2024-08-30T15:59:39.145 INFO:teuthology.orchestra.run.smithi050.stdout:min_compat_client jewel 2024-08-30T15:59:39.145 INFO:teuthology.orchestra.run.smithi050.stdout:require_osd_release reef 2024-08-30T15:59:39.146 INFO:teuthology.orchestra.run.smithi050.stdout:stretch_mode_enabled false 2024-08-30T15:59:39.146 INFO:teuthology.orchestra.run.smithi050.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 19 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-08-30T15:59:39.146 INFO:teuthology.orchestra.run.smithi050.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 28 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 4.98 2024-08-30T15:59:39.146 INFO:teuthology.orchestra.run.smithi050.stdout:max_osd 8 2024-08-30T15:59:39.146 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.50:6812/1957399494 v1:172.21.15.50:6813/1957399494 exists,up 9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4 2024-08-30T15:59:39.146 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1 up in weight 1 up_from 22 up_thru 22 down_at 0 last_clean_interval [0,0) v1:172.21.15.50:6808/3208178756 v1:172.21.15.50:6809/3208178756 exists,up 0fe11a1c-4326-4bfd-a2a4-bff14023f301 2024-08-30T15:59:39.146 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.50:6804/287018800 v1:172.21.15.50:6805/287018800 exists,up a9d5354b-3600-41ed-80c5-a3439da7aca6 2024-08-30T15:59:39.146 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3 up in weight 1 up_from 21 up_thru 24 down_at 0 last_clean_interval [0,0) v1:172.21.15.50:6800/1140717714 v1:172.21.15.50:6801/1140717714 exists,up 78bcc398-a7e4-4efc-9836-1dfb94407c9c 2024-08-30T15:59:39.146 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4 up in weight 1 up_from 20 up_thru 24 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6805/1468073345 v1:172.21.15.94:6806/1468073345 exists,up 53a1060b-7985-4c2c-820e-64b158056113 2024-08-30T15:59:39.146 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5 up in weight 1 up_from 13 up_thru 24 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6809/3876025616 v1:172.21.15.94:6810/3876025616 exists,up 853a03f7-227d-4f98-9e5c-c29896820746 2024-08-30T15:59:39.146 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6801/128519122 v1:172.21.15.94:6802/128519122 exists,up bc978060-bf28-4027-8bba-7d6d8659bfc5 2024-08-30T15:59:39.147 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7 up in weight 1 up_from 21 up_thru 24 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6813/68721277 v1:172.21.15.94:6814/68721277 exists,up 45df5c6f-b124-46d1-9efb-62b78bc584ed 2024-08-30T15:59:39.158 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.50:6812/1957399494 v1:172.21.15.50:6813/1957399494 exists,up 9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4', 'osd.1 up in weight 1 up_from 22 up_thru 22 down_at 0 last_clean_interval [0,0) v1:172.21.15.50:6808/3208178756 v1:172.21.15.50:6809/3208178756 exists,up 0fe11a1c-4326-4bfd-a2a4-bff14023f301', 'osd.2 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.50:6804/287018800 v1:172.21.15.50:6805/287018800 exists,up a9d5354b-3600-41ed-80c5-a3439da7aca6', 'osd.3 up in weight 1 up_from 21 up_thru 24 down_at 0 last_clean_interval [0,0) v1:172.21.15.50:6800/1140717714 v1:172.21.15.50:6801/1140717714 exists,up 78bcc398-a7e4-4efc-9836-1dfb94407c9c', 'osd.4 up in weight 1 up_from 20 up_thru 24 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6805/1468073345 v1:172.21.15.94:6806/1468073345 exists,up 53a1060b-7985-4c2c-820e-64b158056113', 'osd.5 up in weight 1 up_from 13 up_thru 24 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6809/3876025616 v1:172.21.15.94:6810/3876025616 exists,up 853a03f7-227d-4f98-9e5c-c29896820746', 'osd.6 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6801/128519122 v1:172.21.15.94:6802/128519122 exists,up bc978060-bf28-4027-8bba-7d6d8659bfc5', 'osd.7 up in weight 1 up_from 21 up_thru 24 down_at 0 last_clean_interval [0,0) v1:172.21.15.94:6813/68721277 v1:172.21.15.94:6814/68721277 exists,up 45df5c6f-b124-46d1-9efb-62b78bc584ed'] 2024-08-30T15:59:39.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-08-30T15:59:39.337 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mon.a", 2024-08-30T15:59:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster": "ceph", 2024-08-30T15:59:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-08-30T15:59:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "admin_socket_mode": "", 2024-08-30T15:59:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-08-30T15:59:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_client_required": "cephx, none", 2024-08-30T15:59:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_cluster_required": "cephx", 2024-08-30T15:59:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_debug": "true", 2024-08-30T15:59:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-08-30T15:59:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-08-30T15:59:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_service_required": "cephx", 2024-08-30T15:59:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_service_ticket_ttl": "240.000000", 2024-08-30T15:59:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_supported": "cephx", 2024-08-30T15:59:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_aio": "true", 2024-08-30T15:59:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_aio_max_queue_depth": "1024", 2024-08-30T15:59:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_aio_poll_ms": "250", 2024-08-30T15:59:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_aio_reap_max": "16", 2024-08-30T15:59:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_async_discard": "false", 2024-08-30T15:59:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_block_size": "4096", 2024-08-30T15:59:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_debug_aio": "false", 2024-08-30T15:59:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-08-30T15:59:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-08-30T15:59:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_debug_inflight_ios": "false", 2024-08-30T15:59:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_enable_discard": "false", 2024-08-30T15:59:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_flock_retry": "3", 2024-08-30T15:59:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_flock_retry_interval": "0.100000", 2024-08-30T15:59:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_inject_crash": "0", 2024-08-30T15:59:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_inject_crash_flush_delay": "2", 2024-08-30T15:59:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_ioring": "false", 2024-08-30T15:59:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_ioring_hipri": "false", 2024-08-30T15:59:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_ioring_sqthread_poll": "false", 2024-08-30T15:59:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-08-30T15:59:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_read_buffer_alignment": "4096", 2024-08-30T15:59:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-08-30T15:59:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_type": "", 2024-08-30T15:59:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_alloc_size": "1048576", 2024-08-30T15:59:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_allocator": "hybrid", 2024-08-30T15:59:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_buffered_io": "true", 2024-08-30T15:59:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_check_for_zeros": "false", 2024-08-30T15:59:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_check_volume_selector_often": "false", 2024-08-30T15:59:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-08-30T15:59:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_compact_log_sync": "false", 2024-08-30T15:59:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-08-30T15:59:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-08-30T15:59:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_log_compact_min_size": "16777216", 2024-08-30T15:59:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_log_replay_check_allocations": "true", 2024-08-30T15:59:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_max_log_runway": "4194304", 2024-08-30T15:59:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_max_prefetch": "1048576", 2024-08-30T15:59:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_min_flush_size": "524288", 2024-08-30T15:59:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_min_log_runway": "1048576", 2024-08-30T15:59:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_replay_recovery": "false", 2024-08-30T15:59:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-08-30T15:59:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_shared_alloc_size": "65536", 2024-08-30T15:59:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_sync_write": "false", 2024-08-30T15:59:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-08-30T15:59:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-08-30T15:59:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-08-30T15:59:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_allocation_from_file": "true", 2024-08-30T15:59:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_allocator": "hybrid", 2024-08-30T15:59:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-08-30T15:59:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-08-30T15:59:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-08-30T15:59:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-08-30T15:59:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-08-30T15:59:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-08-30T15:59:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blobid_prealloc": "10240", 2024-08-30T15:59:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_create": "true", 2024-08-30T15:59:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_db_create": "false", 2024-08-30T15:59:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_db_path": "", 2024-08-30T15:59:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_db_size": "0", 2024-08-30T15:59:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_path": "", 2024-08-30T15:59:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_preallocate_file": "false", 2024-08-30T15:59:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_size": "107374182400", 2024-08-30T15:59:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_wal_create": "false", 2024-08-30T15:59:39.344 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_wal_path": "", 2024-08-30T15:59:39.344 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_wal_size": "100663296", 2024-08-30T15:59:39.344 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bluefs": "true", 2024-08-30T15:59:39.344 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-08-30T15:59:39.344 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bluefs_env_mirror": "false", 2024-08-30T15:59:39.344 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-08-30T15:59:39.344 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-08-30T15:59:39.344 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-08-30T15:59:39.344 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-08-30T15:59:39.344 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-08-30T15:59:39.344 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-08-30T15:59:39.344 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_autotune": "true", 2024-08-30T15:59:39.344 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-08-30T15:59:39.344 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-08-30T15:59:39.345 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-08-30T15:59:39.345 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-08-30T15:59:39.345 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_size": "0", 2024-08-30T15:59:39.345 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-08-30T15:59:39.345 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-08-30T15:59:39.345 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-08-30T15:59:39.345 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-08-30T15:59:39.345 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_type": "2q", 2024-08-30T15:59:39.345 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-08-30T15:59:39.345 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_clone_cow": "true", 2024-08-30T15:59:39.345 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_algorithm": "snappy", 2024-08-30T15:59:39.345 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_max_blob_size": "0", 2024-08-30T15:59:39.346 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-08-30T15:59:39.346 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-08-30T15:59:39.346 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_min_blob_size": "0", 2024-08-30T15:59:39.346 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-08-30T15:59:39.346 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-08-30T15:59:39.346 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_mode": "none", 2024-08-30T15:59:39.346 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-08-30T15:59:39.346 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_csum_type": "crc32c", 2024-08-30T15:59:39.346 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_enforce_settings": "default", 2024-08-30T15:59:39.346 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_freelist": "false", 2024-08-30T15:59:39.346 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_fsck_abort": "false", 2024-08-30T15:59:39.346 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-08-30T15:59:39.346 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_inject_bug21040": "false", 2024-08-30T15:59:39.347 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-08-30T15:59:39.347 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_inject_read_err": "false", 2024-08-30T15:59:39.347 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_legacy_omap": "false", 2024-08-30T15:59:39.347 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-08-30T15:59:39.347 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-08-30T15:59:39.347 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-08-30T15:59:39.347 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-08-30T15:59:39.347 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_prefill": "0.000000", 2024-08-30T15:59:39.347 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-08-30T15:59:39.347 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-08-30T15:59:39.347 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-08-30T15:59:39.347 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_small_allocations": "0", 2024-08-30T15:59:39.348 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-08-30T15:59:39.348 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_default_buffered_read": "true", 2024-08-30T15:59:39.348 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_default_buffered_write": "false", 2024-08-30T15:59:39.348 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_deferred_batch_ops": "0", 2024-08-30T15:59:39.348 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-08-30T15:59:39.348 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-08-30T15:59:39.348 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-08-30T15:59:39.348 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-08-30T15:59:39.348 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-08-30T15:59:39.348 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-08-30T15:59:39.348 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-08-30T15:59:39.348 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fail_eio": "false", 2024-08-30T15:59:39.348 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-08-30T15:59:39.349 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-08-30T15:59:39.349 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-08-30T15:59:39.349 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-08-30T15:59:39.349 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_mkfs": "true", 2024-08-30T15:59:39.349 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-08-30T15:59:39.349 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_mount": "false", 2024-08-30T15:59:39.349 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-08-30T15:59:39.349 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_umount": "false", 2024-08-30T15:59:39.349 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-08-30T15:59:39.349 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-08-30T15:59:39.349 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-08-30T15:59:39.349 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-08-30T15:59:39.350 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-08-30T15:59:39.350 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-08-30T15:59:39.350 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-08-30T15:59:39.350 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-08-30T15:59:39.350 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_ignore_data_csum": "false", 2024-08-30T15:59:39.350 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-08-30T15:59:39.350 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_kvbackend": "rocksdb", 2024-08-30T15:59:39.350 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-08-30T15:59:39.350 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-08-30T15:59:39.350 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_log_op_age": "5.000000", 2024-08-30T15:59:39.350 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_alloc_size": "0", 2024-08-30T15:59:39.350 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_blob_size": "0", 2024-08-30T15:59:39.350 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-08-30T15:59:39.350 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-08-30T15:59:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_defer_interval": "3.000000", 2024-08-30T15:59:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_deferred_txc": "32", 2024-08-30T15:59:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_min_alloc_size": "0", 2024-08-30T15:59:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-08-30T15:59:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-08-30T15:59:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_nid_prealloc": "1024", 2024-08-30T15:59:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_prefer_deferred_size": "0", 2024-08-30T15:59:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-08-30T15:59:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-08-30T15:59:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_qfsck_on_mount": "true", 2024-08-30T15:59:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_retry_disk_reads": "3", 2024-08-30T15:59:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_rocksdb_cf": "true", 2024-08-30T15:59:39.351 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-08-30T15:59:39.352 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-08-30T15:59:39.352 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_rocksdb_options_annex": "", 2024-08-30T15:59:39.352 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_spdk_coremask": "0x1", 2024-08-30T15:59:39.352 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_spdk_io_sleep": "5", 2024-08-30T15:59:39.352 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_spdk_max_io_completion": "0", 2024-08-30T15:59:39.352 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_spdk_mem": "512", 2024-08-30T15:59:39.352 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_sync_submit_transaction": "false", 2024-08-30T15:59:39.352 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_bytes": "67108864", 2024-08-30T15:59:39.352 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_cost_per_io": "0", 2024-08-30T15:59:39.352 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-08-30T15:59:39.352 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-08-30T15:59:39.352 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-08-30T15:59:39.353 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-08-30T15:59:39.353 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_tracing": "false", 2024-08-30T15:59:39.353 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-08-30T15:59:39.353 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-08-30T15:59:39.353 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_volume_selection_reserved": "0", 2024-08-30T15:59:39.353 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-08-30T15:59:39.353 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-08-30T15:59:39.353 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-08-30T15:59:39.353 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-08-30T15:59:39.353 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-08-30T15:59:39.353 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-08-30T15:59:39.353 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_zero_block_detection": "false", 2024-08-30T15:59:39.353 INFO:teuthology.orchestra.run.smithi050.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-08-30T15:59:39.353 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_action_update_interval": "2", 2024-08-30T15:59:39.354 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-08-30T15:59:39.354 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-08-30T15:59:39.354 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-08-30T15:59:39.354 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-08-30T15:59:39.354 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_mount_timeout": "10", 2024-08-30T15:59:39.354 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-08-30T15:59:39.354 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-08-30T15:59:39.354 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-08-30T15:59:39.354 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-08-30T15:59:39.354 INFO:teuthology.orchestra.run.smithi050.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-08-30T15:59:39.354 INFO:teuthology.orchestra.run.smithi050.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-08-30T15:59:39.354 INFO:teuthology.orchestra.run.smithi050.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-08-30T15:59:39.354 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_cluster_require_signatures": "false", 2024-08-30T15:59:39.355 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_cluster_require_version": "2", 2024-08-30T15:59:39.355 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_require_signatures": "false", 2024-08-30T15:59:39.355 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_require_version": "2", 2024-08-30T15:59:39.355 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_service_require_signatures": "false", 2024-08-30T15:59:39.355 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_service_require_version": "2", 2024-08-30T15:59:39.355 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_sign_messages": "true", 2024-08-30T15:59:39.355 INFO:teuthology.orchestra.run.smithi050.stdout: "chdir": "", 2024-08-30T15:59:39.355 INFO:teuthology.orchestra.run.smithi050.stdout: "client_acl_type": "", 2024-08-30T15:59:39.355 INFO:teuthology.orchestra.run.smithi050.stdout: "client_asio_thread_count": "2", 2024-08-30T15:59:39.355 INFO:teuthology.orchestra.run.smithi050.stdout: "client_cache_mid": "0.750000", 2024-08-30T15:59:39.355 INFO:teuthology.orchestra.run.smithi050.stdout: "client_cache_size": "16384", 2024-08-30T15:59:39.355 INFO:teuthology.orchestra.run.smithi050.stdout: "client_caps_release_delay": "5", 2024-08-30T15:59:39.355 INFO:teuthology.orchestra.run.smithi050.stdout: "client_check_pool_perm": "true", 2024-08-30T15:59:39.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_collect_and_send_global_metrics": "false", 2024-08-30T15:59:39.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_debug_force_sync_read": "false", 2024-08-30T15:59:39.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_debug_getattr_caps": "false", 2024-08-30T15:59:39.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_debug_inject_features": "", 2024-08-30T15:59:39.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_debug_inject_tick_delay": "0", 2024-08-30T15:59:39.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-08-30T15:59:39.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_die_on_failed_remount": "false", 2024-08-30T15:59:39.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_dirsize_rbytes": "true", 2024-08-30T15:59:39.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_force_lazyio": "false", 2024-08-30T15:59:39.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_fs": "", 2024-08-30T15:59:39.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_inject_fixed_oldest_tid": "false", 2024-08-30T15:59:39.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_inject_release_failure": "false", 2024-08-30T15:59:39.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_max_inline_size": "4096", 2024-08-30T15:59:39.357 INFO:teuthology.orchestra.run.smithi050.stdout: "client_max_retries_on_remount_failure": "5", 2024-08-30T15:59:39.357 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mds_namespace": "", 2024-08-30T15:59:39.357 INFO:teuthology.orchestra.run.smithi050.stdout: "client_metadata": "", 2024-08-30T15:59:39.357 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mount_gid": "-1", 2024-08-30T15:59:39.357 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mount_timeout": "300", 2024-08-30T15:59:39.357 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mount_uid": "-1", 2024-08-30T15:59:39.357 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mountpoint": "/", 2024-08-30T15:59:39.357 INFO:teuthology.orchestra.run.smithi050.stdout: "client_notify_timeout": "10", 2024-08-30T15:59:39.357 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc": "true", 2024-08-30T15:59:39.357 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_max_dirty": "104857600", 2024-08-30T15:59:39.357 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_max_dirty_age": "5.000000", 2024-08-30T15:59:39.357 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_max_objects": "1000", 2024-08-30T15:59:39.357 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_size": "209715200", 2024-08-30T15:59:39.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_target_dirty": "8388608", 2024-08-30T15:59:39.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_permissions": "true", 2024-08-30T15:59:39.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_quota": "true", 2024-08-30T15:59:39.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_quota_df": "true", 2024-08-30T15:59:39.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_readahead_max_bytes": "0", 2024-08-30T15:59:39.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_readahead_max_periods": "4", 2024-08-30T15:59:39.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_readahead_min": "131072", 2024-08-30T15:59:39.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_reconnect_stale": "false", 2024-08-30T15:59:39.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_shutdown_timeout": "30", 2024-08-30T15:59:39.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_snapdir": ".snap", 2024-08-30T15:59:39.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_tick_interval": "1", 2024-08-30T15:59:39.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_trace": "", 2024-08-30T15:59:39.359 INFO:teuthology.orchestra.run.smithi050.stdout: "client_try_dentry_invalidate": "false", 2024-08-30T15:59:39.359 INFO:teuthology.orchestra.run.smithi050.stdout: "client_use_faked_inos": "false", 2024-08-30T15:59:39.359 INFO:teuthology.orchestra.run.smithi050.stdout: "client_use_random_mds": "false", 2024-08-30T15:59:39.359 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_graylog": "false", 2024-08-30T15:59:39.359 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-08-30T15:59:39.359 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_graylog_port": "12201", 2024-08-30T15:59:39.359 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_monitors": "default=true", 2024-08-30T15:59:39.359 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_syslog": "false", 2024-08-30T15:59:39.359 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-08-30T15:59:39.359 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_syslog_level": "info", 2024-08-30T15:59:39.359 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "-", 2024-08-30T15:59:39.359 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_network": "", 2024-08-30T15:59:39.359 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_network_interface": "", 2024-08-30T15:59:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "compressor_zlib_isal": "false", 2024-08-30T15:59:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "compressor_zlib_level": "5", 2024-08-30T15:59:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "compressor_zlib_winsize": "-15", 2024-08-30T15:59:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "compressor_zstd_level": "1", 2024-08-30T15:59:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-08-30T15:59:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-08-30T15:59:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_alien_op_num_threads": "6", 2024-08-30T15:59:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_alien_thread_cpu_cores": "", 2024-08-30T15:59:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_osd_obc_lru_size": "10", 2024-08-30T15:59:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-08-30T15:59:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_seastar_smp": "1", 2024-08-30T15:59:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "", 2024-08-30T15:59:39.360 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location_hook": "", 2024-08-30T15:59:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location_hook_timeout": "10", 2024-08-30T15:59:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "daemonize": "false", 2024-08-30T15:59:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "daos_pool": "tank", 2024-08-30T15:59:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-08-30T15:59:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-08-30T15:59:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "dbstore_db_name_prefix": "dbstore", 2024-08-30T15:59:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_alienstore": "0/5", 2024-08-30T15:59:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_allow_any_pool_priority": "false", 2024-08-30T15:59:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_asok": "1/5", 2024-08-30T15:59:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_asok_assert_abort": "false", 2024-08-30T15:59:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_asserts_on_shutdown": "true", 2024-08-30T15:59:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_auth": "20/20", 2024-08-30T15:59:39.361 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_bdev": "1/3", 2024-08-30T15:59:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_bluefs": "1/5", 2024-08-30T15:59:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_bluestore": "1/5", 2024-08-30T15:59:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_buffer": "0/1", 2024-08-30T15:59:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_ceph_exporter": "1/5", 2024-08-30T15:59:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_cephfs_mirror": "0/5", 2024-08-30T15:59:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_cephsqlite": "0/5", 2024-08-30T15:59:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_client": "0/5", 2024-08-30T15:59:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_compressor": "1/5", 2024-08-30T15:59:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_context": "0/1", 2024-08-30T15:59:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_crush": "1/1", 2024-08-30T15:59:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_crypto": "1/5", 2024-08-30T15:59:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_cyanstore": "0/5", 2024-08-30T15:59:39.362 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_deliberately_leak_memory": "false", 2024-08-30T15:59:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_disable_randomized_ping": "false", 2024-08-30T15:59:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_dpdk": "1/5", 2024-08-30T15:59:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_eventtrace": "1/5", 2024-08-30T15:59:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_filer": "0/1", 2024-08-30T15:59:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_filestore": "1/3", 2024-08-30T15:59:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_finisher": "1/1", 2024-08-30T15:59:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_fuse": "1/5", 2024-08-30T15:59:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_heartbeat_testing_span": "0", 2024-08-30T15:59:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_heartbeatmap": "1/5", 2024-08-30T15:59:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_immutable_obj_cache": "0/5", 2024-08-30T15:59:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_javaclient": "1/5", 2024-08-30T15:59:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_journal": "1/3", 2024-08-30T15:59:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_journaler": "0/5", 2024-08-30T15:59:39.363 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_kstore": "1/5", 2024-08-30T15:59:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_leveldb": "4/5", 2024-08-30T15:59:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_lockdep": "0/1", 2024-08-30T15:59:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mclock": "1/5", 2024-08-30T15:59:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds": "1/5", 2024-08-30T15:59:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_balancer": "1/5", 2024-08-30T15:59:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_locker": "1/5", 2024-08-30T15:59:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_log": "1/5", 2024-08-30T15:59:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_log_expire": "1/5", 2024-08-30T15:59:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_migrator": "1/5", 2024-08-30T15:59:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_memstore": "1/5", 2024-08-30T15:59:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mgr": "2/5", 2024-08-30T15:59:39.364 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mgrc": "1/5", 2024-08-30T15:59:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mon": "20/20", 2024-08-30T15:59:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_monc": "0/10", 2024-08-30T15:59:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_ms": "1/1", 2024-08-30T15:59:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_none": "0/5", 2024-08-30T15:59:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_objclass": "0/5", 2024-08-30T15:59:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_objectcacher": "0/5", 2024-08-30T15:59:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_objecter": "0/1", 2024-08-30T15:59:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_optracker": "0/5", 2024-08-30T15:59:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_osd": "1/5", 2024-08-30T15:59:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_paxos": "20/20", 2024-08-30T15:59:39.365 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_perfcounter": "1/5", 2024-08-30T15:59:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_prioritycache": "1/5", 2024-08-30T15:59:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rados": "0/5", 2024-08-30T15:59:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rbd": "0/5", 2024-08-30T15:59:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rbd_mirror": "0/5", 2024-08-30T15:59:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rbd_pwl": "0/5", 2024-08-30T15:59:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rbd_replay": "0/5", 2024-08-30T15:59:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_refs": "0/0", 2024-08-30T15:59:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_reserver": "1/1", 2024-08-30T15:59:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw": "1/5", 2024-08-30T15:59:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_access": "1/5", 2024-08-30T15:59:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_datacache": "1/5", 2024-08-30T15:59:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_dbstore": "1/5", 2024-08-30T15:59:39.366 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_flight": "1/5", 2024-08-30T15:59:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_sync": "1/5", 2024-08-30T15:59:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rocksdb": "4/5", 2024-08-30T15:59:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore": "0/5", 2024-08-30T15:59:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_backref": "0/5", 2024-08-30T15:59:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_cache": "0/5", 2024-08-30T15:59:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_cleaner": "0/5", 2024-08-30T15:59:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_device": "0/5", 2024-08-30T15:59:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_epm": "0/5", 2024-08-30T15:59:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-08-30T15:59:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_journal": "0/5", 2024-08-30T15:59:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_lba": "0/5", 2024-08-30T15:59:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_odata": "0/5", 2024-08-30T15:59:39.367 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_omap": "0/5", 2024-08-30T15:59:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_onode": "0/5", 2024-08-30T15:59:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_t": "0/5", 2024-08-30T15:59:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_tm": "0/5", 2024-08-30T15:59:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_striper": "0/1", 2024-08-30T15:59:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_test": "0/5", 2024-08-30T15:59:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_throttle": "1/1", 2024-08-30T15:59:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_timer": "0/1", 2024-08-30T15:59:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_tp": "0/5", 2024-08-30T15:59:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-08-30T15:59:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "device_failure_prediction_mode": "none", 2024-08-30T15:59:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-08-30T15:59:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-08-30T15:59:39.368 INFO:teuthology.orchestra.run.smithi050.stdout: "err_to_graylog": "false", 2024-08-30T15:59:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "err_to_journald": "false", 2024-08-30T15:59:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "err_to_stderr": "true", 2024-08-30T15:59:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "err_to_syslog": "false", 2024-08-30T15:59:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event_tracing": "false", 2024-08-30T15:59:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_addr": "0.0.0.0", 2024-08-30T15:59:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_http_port": "9926", 2024-08-30T15:59:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_prio_limit": "5", 2024-08-30T15:59:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-08-30T15:59:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_sort_metrics": "true", 2024-08-30T15:59:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_stats_period": "5", 2024-08-30T15:59:39.369 INFO:teuthology.orchestra.run.smithi050.stdout: "fake_statfs_for_testing": "0", 2024-08-30T15:59:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "fatal_signal_handlers": "true", 2024-08-30T15:59:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "filer_max_purge_ops": "10", 2024-08-30T15:59:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "filer_max_truncate_ops": "128", 2024-08-30T15:59:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_apply_finisher_threads": "1", 2024-08-30T15:59:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_blackhole": "false", 2024-08-30T15:59:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_btrfs_clone_range": "true", 2024-08-30T15:59:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_btrfs_snap": "true", 2024-08-30T15:59:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_caller_concurrency": "10", 2024-08-30T15:59:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_collect_device_partition_information": "true", 2024-08-30T15:59:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_commit_timeout": "600.000000", 2024-08-30T15:59:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_debug_inject_read_err": "false", 2024-08-30T15:59:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_debug_omap_check": "false", 2024-08-30T15:59:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_debug_random_read_err": "0.000000", 2024-08-30T15:59:39.370 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_debug_verify_split": "false", 2024-08-30T15:59:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_dump_file": "", 2024-08-30T15:59:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-08-30T15:59:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-08-30T15:59:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fadvise": "true", 2024-08-30T15:59:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fail_eio": "true", 2024-08-30T15:59:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fd_cache_shards": "16", 2024-08-30T15:59:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fd_cache_size": "128", 2024-08-30T15:59:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fiemap": "false", 2024-08-30T15:59:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fiemap_threshold": "4096", 2024-08-30T15:59:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-08-30T15:59:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_index_retry_probability": "0.000000", 2024-08-30T15:59:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_inject_stall": "0", 2024-08-30T15:59:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_journal_parallel": "false", 2024-08-30T15:59:39.371 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_journal_trailing": "false", 2024-08-30T15:59:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_journal_writeahead": "false", 2024-08-30T15:59:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_kill_at": "0", 2024-08-30T15:59:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-08-30T15:59:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattr_size": "0", 2024-08-30T15:59:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-08-30T15:59:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-08-30T15:59:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-08-30T15:59:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattrs": "0", 2024-08-30T15:59:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-08-30T15:59:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattrs_other": "2", 2024-08-30T15:59:39.372 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-08-30T15:59:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_sync_interval": "5.000000", 2024-08-30T15:59:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_xattr_value_size": "0", 2024-08-30T15:59:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-08-30T15:59:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-08-30T15:59:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-08-30T15:59:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_merge_threshold": "-10", 2024-08-30T15:59:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_min_sync_interval": "0.010000", 2024-08-30T15:59:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_odsync_write": "false", 2024-08-30T15:59:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_omap_backend": "rocksdb", 2024-08-30T15:59:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_omap_backend_path": "", 2024-08-30T15:59:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_omap_header_cache_size": "1024", 2024-08-30T15:59:39.373 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_ondisk_finisher_threads": "1", 2024-08-30T15:59:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-08-30T15:59:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_op_thread_timeout": "60", 2024-08-30T15:59:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_op_threads": "2", 2024-08-30T15:59:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_punch_hole": "false", 2024-08-30T15:59:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-08-30T15:59:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-08-30T15:59:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-08-30T15:59:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-08-30T15:59:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-08-30T15:59:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_bytes": "104857600", 2024-08-30T15:59:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-08-30T15:59:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-08-30T15:59:39.374 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-08-30T15:59:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_ops": "50", 2024-08-30T15:59:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-08-30T15:59:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_seek_data_hole": "false", 2024-08-30T15:59:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_sloppy_crc": "false", 2024-08-30T15:59:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-08-30T15:59:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_splice": "false", 2024-08-30T15:59:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_split_multiple": "2", 2024-08-30T15:59:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_split_rand_factor": "20", 2024-08-30T15:59:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_update_to": "1000", 2024-08-30T15:59:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-08-30T15:59:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-08-30T15:59:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-08-30T15:59:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-08-30T15:59:39.375 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-08-30T15:59:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-08-30T15:59:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_enable": "true", 2024-08-30T15:59:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-08-30T15:59:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-08-30T15:59:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-08-30T15:59:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-08-30T15:59:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-08-30T15:59:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-08-30T15:59:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_xfs_extsize": "false", 2024-08-30T15:59:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_zfs_snap": "false", 2024-08-30T15:59:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "fio_dir": "/tmp/fio", 2024-08-30T15:59:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "9d72e2e2-0554-42af-8e62-9b1a3a804112", 2024-08-30T15:59:39.376 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_allow_other": "true", 2024-08-30T15:59:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_atomic_o_trunc": "true", 2024-08-30T15:59:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_big_writes": "true", 2024-08-30T15:59:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_debug": "false", 2024-08-30T15:59:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_default_permissions": "false", 2024-08-30T15:59:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_disable_pagecache": "false", 2024-08-30T15:59:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_max_write": "0", 2024-08-30T15:59:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_multithreaded": "true", 2024-08-30T15:59:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_require_active_mds": "true", 2024-08-30T15:59:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_set_user_groups": "true", 2024-08-30T15:59:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_splice_move": "true", 2024-08-30T15:59:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_splice_read": "true", 2024-08-30T15:59:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_splice_write": "true", 2024-08-30T15:59:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_syncfs_on_mksnap": "true", 2024-08-30T15:59:39.377 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_use_invalidate_cb": "true", 2024-08-30T15:59:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-08-30T15:59:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "gss_target_name": "ceph", 2024-08-30T15:59:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_file": "", 2024-08-30T15:59:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_inject_failure": "0", 2024-08-30T15:59:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_interval": "5", 2024-08-30T15:59:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "host": "smithi050", 2024-08-30T15:59:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-08-30T15:59:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-08-30T15:59:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_max_size": "1073741824", 2024-08-30T15:59:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-08-30T15:59:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-08-30T15:59:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-08-30T15:59:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-08-30T15:59:39.378 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-08-30T15:59:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-08-30T15:59:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-08-30T15:59:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-08-30T15:59:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-08-30T15:59:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_watermark": "0.900000", 2024-08-30T15:59:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "inject_early_sigterm": "false", 2024-08-30T15:59:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "jaeger_agent_port": "6799", 2024-08-30T15:59:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "jaeger_tracing_enable": "false", 2024-08-30T15:59:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_aio": "true", 2024-08-30T15:59:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_align_min_size": "65536", 2024-08-30T15:59:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_block_align": "true", 2024-08-30T15:59:39.379 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_block_size": "4096", 2024-08-30T15:59:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_dio": "true", 2024-08-30T15:59:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_discard": "false", 2024-08-30T15:59:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_force_aio": "false", 2024-08-30T15:59:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_ignore_corruption": "false", 2024-08-30T15:59:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_max_write_bytes": "10485760", 2024-08-30T15:59:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_max_write_entries": "100", 2024-08-30T15:59:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_replay_from": "0", 2024-08-30T15:59:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_throttle_high_multiple": "0.000000", 2024-08-30T15:59:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-08-30T15:59:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-08-30T15:59:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_throttle_max_multiple": "0.000000", 2024-08-30T15:59:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_write_header_frequency": "0", 2024-08-30T15:59:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_zero_on_create": "false", 2024-08-30T15:59:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "journaler_prefetch_periods": "10", 2024-08-30T15:59:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "journaler_prezero_periods": "5", 2024-08-30T15:59:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "journaler_write_head_interval": "15", 2024-08-30T15:59:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-08-30T15:59:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "keyfile": "", 2024-08-30T15:59:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-08-30T15:59:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_backend": "rocksdb", 2024-08-30T15:59:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_default_stripe_size": "65536", 2024-08-30T15:59:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_fsck_on_mount": "false", 2024-08-30T15:59:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_fsck_on_mount_deep": "true", 2024-08-30T15:59:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_max_bytes": "67108864", 2024-08-30T15:59:39.381 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_max_ops": "512", 2024-08-30T15:59:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_nid_prealloc": "1024", 2024-08-30T15:59:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_onode_map_size": "1024", 2024-08-30T15:59:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-08-30T15:59:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_sync_submit_transaction": "false", 2024-08-30T15:59:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_sync_transaction": "false", 2024-08-30T15:59:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "leveldb_block_size": "65536", 2024-08-30T15:59:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "leveldb_bloom_size": "0", 2024-08-30T15:59:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "leveldb_cache_size": "536870912", 2024-08-30T15:59:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "leveldb_compact_on_mount": "false", 2024-08-30T15:59:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "leveldb_compression": "false", 2024-08-30T15:59:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "leveldb_log": "", 2024-08-30T15:59:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "leveldb_log_to_ceph_log": "true", 2024-08-30T15:59:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "leveldb_max_open_files": "0", 2024-08-30T15:59:39.382 INFO:teuthology.orchestra.run.smithi050.stdout: "leveldb_paranoid": "false", 2024-08-30T15:59:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "leveldb_write_buffer_size": "33554432", 2024-08-30T15:59:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "librados_thread_count": "2", 2024-08-30T15:59:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "lockdep": "false", 2024-08-30T15:59:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "lockdep_force_backtrace": "false", 2024-08-30T15:59:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "log_coarse_timestamps": "true", 2024-08-30T15:59:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-08-30T15:59:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "log_flush_on_exit": "false", 2024-08-30T15:59:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "log_graylog_host": "127.0.0.1", 2024-08-30T15:59:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "log_graylog_port": "12201", 2024-08-30T15:59:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "log_max_new": "1000", 2024-08-30T15:59:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "log_max_recent": "10000", 2024-08-30T15:59:39.383 INFO:teuthology.orchestra.run.smithi050.stdout: "log_stderr_prefix": "", 2024-08-30T15:59:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "log_stop_at_utilization": "0.970000", 2024-08-30T15:59:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": "true", 2024-08-30T15:59:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_graylog": "false", 2024-08-30T15:59:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_journald": "false", 2024-08-30T15:59:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_stderr": "false", 2024-08-30T15:59:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_syslog": "false", 2024-08-30T15:59:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "max_rotating_auth_attempts": "10", 2024-08-30T15:59:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-08-30T15:59:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_action_on_write_error": "1", 2024-08-30T15:59:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_alternate_name_max": "8192", 2024-08-30T15:59:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_asio_thread_count": "2", 2024-08-30T15:59:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_export_pin": "true", 2024-08-30T15:59:39.384 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_fragment_dirs": "true", 2024-08-30T15:59:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-08-30T15:59:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_fragment_interval": "5", 2024-08-30T15:59:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_fragment_size_max": "100000", 2024-08-30T15:59:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_idle_threshold": "0.000000", 2024-08-30T15:59:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_interval": "10", 2024-08-30T15:59:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_max": "-1", 2024-08-30T15:59:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_max_until": "-1", 2024-08-30T15:59:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_merge_size": "50", 2024-08-30T15:59:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_midchunk": "0.300000", 2024-08-30T15:59:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_min_rebalance": "0.100000", 2024-08-30T15:59:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_min_start": "0.200000", 2024-08-30T15:59:39.385 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_minchunk": "0.001000", 2024-08-30T15:59:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_mode": "0", 2024-08-30T15:59:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_need_max": "1.200000", 2024-08-30T15:59:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_need_min": "0.800000", 2024-08-30T15:59:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-08-30T15:59:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_sample_interval": "3.000000", 2024-08-30T15:59:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_split_bits": "3", 2024-08-30T15:59:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_split_rd": "25000.000000", 2024-08-30T15:59:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_split_size": "10000", 2024-08-30T15:59:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_split_wr": "10000.000000", 2024-08-30T15:59:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_target_decay": "10.000000", 2024-08-30T15:59:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-08-30T15:59:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_beacon_grace": "15.000000", 2024-08-30T15:59:39.386 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_beacon_interval": "4.000000", 2024-08-30T15:59:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_beacon_mon_down_grace": "60", 2024-08-30T15:59:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_memory_limit": "4294967296", 2024-08-30T15:59:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_mid": "0.700000", 2024-08-30T15:59:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_release_free_interval": "10", 2024-08-30T15:59:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_reservation": "0.050000", 2024-08-30T15:59:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-08-30T15:59:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_trim_interval": "1", 2024-08-30T15:59:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_trim_threshold": "262144", 2024-08-30T15:59:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-08-30T15:59:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-08-30T15:59:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_client_delegate_inos_pct": "50", 2024-08-30T15:59:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_client_prealloc_inos": "1000", 2024-08-30T15:59:39.387 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-08-30T15:59:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_connect_bootstrapping": "false", 2024-08-30T15:59:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_damage_table_max_entries": "10000", 2024-08-30T15:59:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-08-30T15:59:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_debug_auth_pins": "false", 2024-08-30T15:59:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_debug_frag": "false", 2024-08-30T15:59:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_debug_scatterstat": "false", 2024-08-30T15:59:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_debug_subtrees": "false", 2024-08-30T15:59:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_decay_halflife": "5.000000", 2024-08-30T15:59:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_default_dir_hash": "2", 2024-08-30T15:59:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_defer_session_stale": "true", 2024-08-30T15:59:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_deny_all_reconnect": "false", 2024-08-30T15:59:39.388 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dir_keys_per_op": "16384", 2024-08-30T15:59:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dir_max_commit_size": "10", 2024-08-30T15:59:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dir_max_entries": "0", 2024-08-30T15:59:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dir_prefetch": "true", 2024-08-30T15:59:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dirstat_min_interval": "1.000000", 2024-08-30T15:59:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dump_cache_after_rejoin": "false", 2024-08-30T15:59:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dump_cache_on_map": "false", 2024-08-30T15:59:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dump_cache_threshold_file": "0", 2024-08-30T15:59:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-08-30T15:59:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_early_reply": "true", 2024-08-30T15:59:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_enable_op_tracker": "true", 2024-08-30T15:59:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_enforce_unique_name": "true", 2024-08-30T15:59:39.389 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_export_ephemeral_distributed": "true", 2024-08-30T15:59:39.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-08-30T15:59:39.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_export_ephemeral_random": "true", 2024-08-30T15:59:39.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-08-30T15:59:39.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-08-30T15:59:39.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_forward_all_requests_to_auth": "false", 2024-08-30T15:59:39.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-08-30T15:59:39.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-08-30T15:59:39.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-08-30T15:59:39.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-08-30T15:59:39.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_health_cache_threshold": "1.500000", 2024-08-30T15:59:39.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_health_summarize_threshold": "10", 2024-08-30T15:59:39.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_heartbeat_grace": "15.000000", 2024-08-30T15:59:39.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_heartbeat_reset_grace": "1000", 2024-08-30T15:59:39.390 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_health_dummy": "false", 2024-08-30T15:59:39.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-08-30T15:59:39.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_migrator_session_race": "false", 2024-08-30T15:59:39.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-08-30T15:59:39.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-08-30T15:59:39.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-08-30T15:59:39.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_join_fs": "", 2024-08-30T15:59:39.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_journal_format": "1", 2024-08-30T15:59:39.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-08-30T15:59:39.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_create_at": "0", 2024-08-30T15:59:39.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_export_at": "0", 2024-08-30T15:59:39.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_import_at": "0", 2024-08-30T15:59:39.391 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_journal_at": "0", 2024-08-30T15:59:39.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_journal_expire_at": "0", 2024-08-30T15:59:39.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_journal_replay_at": "0", 2024-08-30T15:59:39.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_link_at": "0", 2024-08-30T15:59:39.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_mdstable_at": "0", 2024-08-30T15:59:39.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_openc_at": "0", 2024-08-30T15:59:39.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_rename_at": "0", 2024-08-30T15:59:39.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_events_per_segment": "1024", 2024-08-30T15:59:39.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_max_events": "-1", 2024-08-30T15:59:39.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_max_segments": "128", 2024-08-30T15:59:39.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_pause": "false", 2024-08-30T15:59:39.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_segment_size": "0", 2024-08-30T15:59:39.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_skip_corrupt_events": "false", 2024-08-30T15:59:39.392 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_warn_factor": "2.000000", 2024-08-30T15:59:39.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_caps_per_client": "1048576", 2024-08-30T15:59:39.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_completed_flushes": "100000", 2024-08-30T15:59:39.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_completed_requests": "100000", 2024-08-30T15:59:39.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_export_size": "20971520", 2024-08-30T15:59:39.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_file_recover": "32", 2024-08-30T15:59:39.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_purge_files": "64", 2024-08-30T15:59:39.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_purge_ops": "8192", 2024-08-30T15:59:39.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-08-30T15:59:39.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-08-30T15:59:39.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_snaps_per_dir": "100", 2024-08-30T15:59:39.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_metrics_update_interval": "2", 2024-08-30T15:59:39.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_min_caps_per_client": "100", 2024-08-30T15:59:39.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_min_caps_working_set": "10000", 2024-08-30T15:59:39.393 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_numa_node": "-1", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_complaint_time": "30.000000", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_history_duration": "600", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_history_size": "20", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_history_slow_op_size": "20", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_history_slow_op_threshold": "10", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_log_threshold": "5", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_ping_grace": "15", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_ping_interval": "5", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_max_caps": "30000", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-08-30T15:59:39.394 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_max_decay_threshold": "131072", 2024-08-30T15:59:39.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-08-30T15:59:39.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_warning_threshold": "262144", 2024-08-30T15:59:39.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_reconnect_timeout": "45.000000", 2024-08-30T15:59:39.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_replay_interval": "1.000000", 2024-08-30T15:59:39.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-08-30T15:59:39.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-08-30T15:59:39.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_root_ino_gid": "0", 2024-08-30T15:59:39.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_root_ino_uid": "0", 2024-08-30T15:59:39.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-08-30T15:59:39.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_blocklist_on_evict": "true", 2024-08-30T15:59:39.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_blocklist_on_timeout": "true", 2024-08-30T15:59:39.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-08-30T15:59:39.395 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-08-30T15:59:39.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-08-30T15:59:39.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-08-30T15:59:39.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-08-30T15:59:39.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_metadata_threshold": "16777216", 2024-08-30T15:59:39.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-08-30T15:59:39.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_shutdown_check": "0", 2024-08-30T15:59:39.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_skip_ino": "0", 2024-08-30T15:59:39.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_sleep_rank_change": "0.000000", 2024-08-30T15:59:39.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_snap_max_uid": "4294967294", 2024-08-30T15:59:39.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_snap_min_uid": "0", 2024-08-30T15:59:39.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_snap_rstat": "false", 2024-08-30T15:59:39.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_standby_replay_damaged": "false", 2024-08-30T15:59:39.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_symlink_recovery": "true", 2024-08-30T15:59:39.397 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_task_status_update_interval": "2.000000", 2024-08-30T15:59:39.397 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_thrash_exports": "0", 2024-08-30T15:59:39.397 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_thrash_fragments": "0", 2024-08-30T15:59:39.397 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_tick_interval": "5.000000", 2024-08-30T15:59:39.397 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_valgrind_exit": "false", 2024-08-30T15:59:39.397 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_verify_backtrace": "1", 2024-08-30T15:59:39.397 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_verify_scatter": "false", 2024-08-30T15:59:39.397 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_wipe_ino_prealloc": "false", 2024-08-30T15:59:39.397 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_wipe_sessions": "false", 2024-08-30T15:59:39.397 INFO:teuthology.orchestra.run.smithi050.stdout: "mempool_debug": "false", 2024-08-30T15:59:39.397 INFO:teuthology.orchestra.run.smithi050.stdout: "memstore_debug_omit_block_device_write": "false", 2024-08-30T15:59:39.397 INFO:teuthology.orchestra.run.smithi050.stdout: "memstore_device_bytes": "1073741824", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "memstore_page_set": "false", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "memstore_page_size": "65536", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_client_bytes": "134217728", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_client_messages": "512", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_connect_retry_interval": "1.000000", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_disabled_modules": "", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_max_pg_num_change": "128", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_mds_bytes": "134217728", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_mds_messages": "128", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-08-30T15:59:39.398 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_mon_bytes": "134217728", 2024-08-30T15:59:39.399 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_mon_messages": "128", 2024-08-30T15:59:39.399 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_osd_bytes": "536870912", 2024-08-30T15:59:39.399 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_osd_messages": "8192", 2024-08-30T15:59:39.399 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_pool": "true", 2024-08-30T15:59:39.399 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_service_beacon_grace": "60.000000", 2024-08-30T15:59:39.399 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_standby_modules": "true", 2024-08-30T15:59:39.399 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_stats_period": "5", 2024-08-30T15:59:39.399 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_stats_threshold": "5", 2024-08-30T15:59:39.399 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_tick_period": "2", 2024-08-30T15:59:39.399 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-08-30T15:59:39.399 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_accept_timeout_factor": "2.000000", 2024-08-30T15:59:39.399 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_allow_pool_delete": "true", 2024-08-30T15:59:39.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_allow_pool_size_one": "true", 2024-08-30T15:59:39.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_auth_validate_all_caps": "true", 2024-08-30T15:59:39.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-08-30T15:59:39.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-08-30T15:59:39.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_bytes": "104857600", 2024-08-30T15:59:39.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_directed_command_retry": "5", 2024-08-30T15:59:39.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_interval": "3.000000", 2024-08-30T15:59:39.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-08-30T15:59:39.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-08-30T15:59:39.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-08-30T15:59:39.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_parallel": "3", 2024-08-30T15:59:39.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_log_interval": "1.000000", 2024-08-30T15:59:39.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-08-30T15:59:39.400 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_ping_interval": "10.000000", 2024-08-30T15:59:39.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_ping_timeout": "30.000000", 2024-08-30T15:59:39.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_clock_drift_allowed": "1.000000", 2024-08-30T15:59:39.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-08-30T15:59:39.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-08-30T15:59:39.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_file_level": "debug", 2024-08-30T15:59:39.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_file": "true", 2024-08-30T15:59:39.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_graylog": "false", 2024-08-30T15:59:39.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-08-30T15:59:39.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-08-30T15:59:39.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_journald": "false", 2024-08-30T15:59:39.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_stderr": "false", 2024-08-30T15:59:39.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-08-30T15:59:39.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-08-30T15:59:39.401 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-08-30T15:59:39.402 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_compact_on_bootstrap": "false", 2024-08-30T15:59:39.402 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_compact_on_start": "false", 2024-08-30T15:59:39.402 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_compact_on_trim": "true", 2024-08-30T15:59:39.402 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_con_tracker_persist_interval": "10", 2024-08-30T15:59:39.402 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_con_tracker_score_halflife": "43200", 2024-08-30T15:59:39.402 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_config_key_max_entry_size": "65536", 2024-08-30T15:59:39.402 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cpu_threads": "4", 2024-08-30T15:59:39.402 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_crush_min_required_version": "hammer", 2024-08-30T15:59:39.402 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_daemon_bytes": "419430400", 2024-08-30T15:59:39.402 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-08-30T15:59:39.402 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_data_avail_crit": "5", 2024-08-30T15:59:39.402 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_data_avail_warn": "5", 2024-08-30T15:59:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_data_size_warn": "16106127360", 2024-08-30T15:59:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_block_osdmap_trim": "false", 2024-08-30T15:59:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-08-30T15:59:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_dump_json": "false", 2024-08-30T15:59:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-08-30T15:59:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_dump_transactions": "false", 2024-08-30T15:59:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_extra_checks": "false", 2024-08-30T15:59:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-08-30T15:59:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-08-30T15:59:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_no_require_quincy": "false", 2024-08-30T15:59:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_no_require_reef": "false", 2024-08-30T15:59:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-08-30T15:59:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_delta_reset_interval": "10.000000", 2024-08-30T15:59:39.403 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_dns_srv_name": "ceph-mon", 2024-08-30T15:59:39.404 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_down_mkfs_grace": "120", 2024-08-30T15:59:39.404 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_election_default_strategy": "1", 2024-08-30T15:59:39.404 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_election_timeout": "5.000000", 2024-08-30T15:59:39.404 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-08-30T15:59:39.404 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_elector_ping_divisor": "2", 2024-08-30T15:59:39.404 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_elector_ping_timeout": "2.000000", 2024-08-30T15:59:39.404 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_enable_op_tracker": "true", 2024-08-30T15:59:39.404 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_fake_pool_delete": "false", 2024-08-30T15:59:39.404 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_force_quorum_join": "false", 2024-08-30T15:59:39.404 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_globalid_prealloc": "10000", 2024-08-30T15:59:39.404 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_detail_to_clog": "false", 2024-08-30T15:59:39.404 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_log_update_period": "5", 2024-08-30T15:59:39.404 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_max_detail": "50", 2024-08-30T15:59:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_to_clog": "true", 2024-08-30T15:59:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_to_clog_interval": "600", 2024-08-30T15:59:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-08-30T15:59:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_host": "[v1:172.21.15.50:6789],[v1:172.21.15.50:6790],[v1:172.21.15.94:6789]", 2024-08-30T15:59:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_host_override": "", 2024-08-30T15:59:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_initial_members": "", 2024-08-30T15:59:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-08-30T15:59:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-08-30T15:59:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-08-30T15:59:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-08-30T15:59:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_keyvaluedb": "rocksdb", 2024-08-30T15:59:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_lease": "5.000000", 2024-08-30T15:59:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-08-30T15:59:39.405 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-08-30T15:59:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_log_full_interval": "50", 2024-08-30T15:59:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_log_max": "10000", 2024-08-30T15:59:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_log_max_summary": "50", 2024-08-30T15:59:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_log_entries_per_event": "4096", 2024-08-30T15:59:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_log_epochs": "500", 2024-08-30T15:59:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_mdsmap_epochs": "500", 2024-08-30T15:59:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_mgrmap_epochs": "500", 2024-08-30T15:59:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_osd": "10000", 2024-08-30T15:59:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_pg_per_osd": "10000", 2024-08-30T15:59:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_pool_pg_num": "65536", 2024-08-30T15:59:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-08-30T15:59:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-08-30T15:59:39.407 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mds_force_trim_to": "0", 2024-08-30T15:59:39.407 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mds_skip_sanity": "false", 2024-08-30T15:59:39.407 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_memory_autotune": "true", 2024-08-30T15:59:39.407 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_memory_target": "2147483648", 2024-08-30T15:59:39.407 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_beacon_grace": "30", 2024-08-30T15:59:39.407 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-08-30T15:59:39.407 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_digest_period": "5", 2024-08-30T15:59:39.407 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_inactive_grace": "60", 2024-08-30T15:59:39.407 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_mkfs_grace": "240", 2024-08-30T15:59:39.407 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-08-30T15:59:39.407 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_min_osdmap_epochs": "50", 2024-08-30T15:59:39.407 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_complaint_time": "30", 2024-08-30T15:59:39.407 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_history_duration": "600", 2024-08-30T15:59:39.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_history_size": "20", 2024-08-30T15:59:39.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_history_slow_op_size": "20", 2024-08-30T15:59:39.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_history_slow_op_threshold": "10", 2024-08-30T15:59:39.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_log_threshold": "5", 2024-08-30T15:59:39.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-08-30T15:59:39.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-08-30T15:59:39.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-08-30T15:59:39.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_auto_mark_in": "false", 2024-08-30T15:59:39.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_auto_mark_new_in": "true", 2024-08-30T15:59:39.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-08-30T15:59:39.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-08-30T15:59:39.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_cache_size": "500", 2024-08-30T15:59:39.408 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_cache_size_min": "134217728", 2024-08-30T15:59:39.409 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_crush_smoke_test": "false", 2024-08-30T15:59:39.409 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_destroyed_out_interval": "600", 2024-08-30T15:59:39.409 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_down_out_interval": "600", 2024-08-30T15:59:39.409 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-08-30T15:59:39.409 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-08-30T15:59:39.409 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_force_trim_to": "0", 2024-08-30T15:59:39.409 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_full_ratio": "0.950000", 2024-08-30T15:59:39.409 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-08-30T15:59:39.409 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_laggy_halflife": "3600", 2024-08-30T15:59:39.409 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_laggy_max_interval": "300", 2024-08-30T15:59:39.409 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_laggy_weight": "0.300000", 2024-08-30T15:59:39.409 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-08-30T15:59:39.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_max_creating_pgs": "1024", 2024-08-30T15:59:39.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_max_initial_pgs": "1024", 2024-08-30T15:59:39.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_min_down_reporters": "2", 2024-08-30T15:59:39.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-08-30T15:59:39.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-08-30T15:59:39.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-08-30T15:59:39.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_prime_pg_temp": "true", 2024-08-30T15:59:39.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-08-30T15:59:39.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-08-30T15:59:39.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_report_timeout": "900", 2024-08-30T15:59:39.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-08-30T15:59:39.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-08-30T15:59:39.410 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_warn_num_repaired": "10", 2024-08-30T15:59:39.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_warn_op_age": "32.000000", 2024-08-30T15:59:39.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-08-30T15:59:39.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osdmap_full_prune_interval": "2", 2024-08-30T15:59:39.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osdmap_full_prune_min": "15", 2024-08-30T15:59:39.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-08-30T15:59:39.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-08-30T15:59:39.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_stuck_threshold": "60", 2024-08-30T15:59:39.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-08-30T15:59:39.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_warn_min_objects": "10000", 2024-08-30T15:59:39.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_warn_min_per_osd": "0", 2024-08-30T15:59:39.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-08-30T15:59:39.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pool_quota_crit_threshold": "0", 2024-08-30T15:59:39.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pool_quota_warn_threshold": "0", 2024-08-30T15:59:39.411 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_probe_timeout": "2.000000", 2024-08-30T15:59:39.412 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_reweight_max_change": "0.050000", 2024-08-30T15:59:39.412 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_reweight_max_osds": "4", 2024-08-30T15:59:39.412 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-08-30T15:59:39.412 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-08-30T15:59:39.412 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-08-30T15:59:39.412 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-08-30T15:59:39.412 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-08-30T15:59:39.412 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_interval": "300", 2024-08-30T15:59:39.412 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_max_keys": "100", 2024-08-30T15:59:39.412 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_timeout": "300", 2024-08-30T15:59:39.412 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_session_timeout": "300", 2024-08-30T15:59:39.412 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_smart_report_timeout": "5", 2024-08-30T15:59:39.412 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stat_smooth_intervals": "6", 2024-08-30T15:59:39.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-08-30T15:59:39.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stretch_pool_min_size": "2", 2024-08-30T15:59:39.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stretch_pool_size": "4", 2024-08-30T15:59:39.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-08-30T15:59:39.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_subscribe_interval": "86400.000000", 2024-08-30T15:59:39.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_debug": "false", 2024-08-30T15:59:39.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_max_payload_keys": "2000", 2024-08-30T15:59:39.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_max_payload_size": "1048576", 2024-08-30T15:59:39.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_provider_kill_at": "0", 2024-08-30T15:59:39.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_requester_kill_at": "0", 2024-08-30T15:59:39.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_timeout": "60.000000", 2024-08-30T15:59:39.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_target_pg_per_osd": "100", 2024-08-30T15:59:39.413 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_tick_interval": "5", 2024-08-30T15:59:39.414 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_timecheck_interval": "300.000000", 2024-08-30T15:59:39.414 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-08-30T15:59:39.414 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_use_min_delay_socket": "false", 2024-08-30T15:59:39.414 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_older_version_delay": "604800", 2024-08-30T15:59:39.414 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-08-30T15:59:39.414 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-08-30T15:59:39.414 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-08-30T15:59:39.414 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_filestore_osds": "false", 2024-08-30T15:59:39.414 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-08-30T15:59:39.414 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-08-30T15:59:39.414 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-08-30T15:59:39.414 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_misplaced": "false", 2024-08-30T15:59:39.414 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-08-30T15:59:39.415 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_older_version": "true", 2024-08-30T15:59:39.415 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-08-30T15:59:39.415 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_pool_no_app": "true", 2024-08-30T15:59:39.415 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-08-30T15:59:39.415 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-08-30T15:59:39.415 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-08-30T15:59:39.415 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-08-30T15:59:39.415 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_too_few_osds": "false", 2024-08-30T15:59:39.415 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-08-30T15:59:39.415 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-08-30T15:59:39.415 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": "", 2024-08-30T15:59:39.415 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-08-30T15:59:39.416 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-08-30T15:59:39.416 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-08-30T15:59:39.416 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-08-30T15:59:39.416 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-08-30T15:59:39.416 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-08-30T15:59:39.416 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_tracing_enabled": "false", 2024-08-30T15:59:39.416 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_op_threads": "3", 2024-08-30T15:59:39.416 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_buffer_size": "131072", 2024-08-30T15:59:39.416 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_cm": "false", 2024-08-30T15:59:39.416 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_device_name": "", 2024-08-30T15:59:39.416 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_dscp": "96", 2024-08-30T15:59:39.416 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-08-30T15:59:39.416 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_gid_idx": "0", 2024-08-30T15:59:39.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_local_gid": "", 2024-08-30T15:59:39.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_polling_us": "1000", 2024-08-30T15:59:39.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_port_num": "1", 2024-08-30T15:59:39.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-08-30T15:59:39.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-08-30T15:59:39.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_roce_ver": "1", 2024-08-30T15:59:39.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_send_buffers": "1024", 2024-08-30T15:59:39.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_sl": "3", 2024-08-30T15:59:39.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_support_srq": "true", 2024-08-30T15:59:39.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_type": "ib", 2024-08-30T15:59:39.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_reap_threshold": "5", 2024-08-30T15:59:39.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_before_connect": "false", 2024-08-30T15:59:39.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_ipv4": "true", 2024-08-30T15:59:39.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_ipv6": "false", 2024-08-30T15:59:39.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_msgr1": "true", 2024-08-30T15:59:39.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_msgr2": "false", 2024-08-30T15:59:39.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_port_max": "7568", 2024-08-30T15:59:39.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_port_min": "6800", 2024-08-30T15:59:39.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_prefer_ipv4": "false", 2024-08-30T15:59:39.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_retry_count": "3", 2024-08-30T15:59:39.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_retry_delay": "5", 2024-08-30T15:59:39.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_client": "false", 2024-08-30T15:59:39.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_mds": "false", 2024-08-30T15:59:39.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_mgr": "false", 2024-08-30T15:59:39.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_mon": "false", 2024-08-30T15:59:39.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_osd": "false", 2024-08-30T15:59:39.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_client_mode": "crc secure", 2024-08-30T15:59:39.419 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_cluster_mode": "crc secure", 2024-08-30T15:59:39.419 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_cluster_type": "", 2024-08-30T15:59:39.419 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_compress_secure": "false", 2024-08-30T15:59:39.419 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_connection_idle_timeout": "900", 2024-08-30T15:59:39.419 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_connection_ready_timeout": "10", 2024-08-30T15:59:39.419 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_crc_data": "true", 2024-08-30T15:59:39.419 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_crc_header": "true", 2024-08-30T15:59:39.419 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_bad_msg": "false", 2024-08-30T15:59:39.419 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_bug": "true", 2024-08-30T15:59:39.419 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_old_message": "true", 2024-08-30T15:59:39.419 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_skipped_message": "false", 2024-08-30T15:59:39.419 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_unhandled_msg": "false", 2024-08-30T15:59:39.419 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-08-30T15:59:39.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_coremask": "0xF", 2024-08-30T15:59:39.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-08-30T15:59:39.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_devs_allowlist": "", 2024-08-30T15:59:39.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_enable_tso": "true", 2024-08-30T15:59:39.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-08-30T15:59:39.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-08-30T15:59:39.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_hugepages": "", 2024-08-30T15:59:39.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_hw_flow_control": "true", 2024-08-30T15:59:39.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-08-30T15:59:39.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_lro": "true", 2024-08-30T15:59:39.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_memory_channel": "4", 2024-08-30T15:59:39.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-08-30T15:59:39.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_pmd": "", 2024-08-30T15:59:39.421 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_port_id": "0", 2024-08-30T15:59:39.421 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-08-30T15:59:39.421 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dump_corrupt_message_level": "1", 2024-08-30T15:59:39.421 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dump_on_send": "false", 2024-08-30T15:59:39.421 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_initial_backoff": "0.200000", 2024-08-30T15:59:39.421 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_delay_max": "1.000000", 2024-08-30T15:59:39.421 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_delay_probability": "0.000000", 2024-08-30T15:59:39.421 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_delay_type": "", 2024-08-30T15:59:39.421 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_internal_delays": "0.000000", 2024-08-30T15:59:39.421 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_network_congestion": "0", 2024-08-30T15:59:39.421 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_socket_failures": "5000", 2024-08-30T15:59:39.421 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_learn_addr_from_peer": "true", 2024-08-30T15:59:39.421 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_max_accept_failures": "4", 2024-08-30T15:59:39.422 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_max_backoff": "15.000000", 2024-08-30T15:59:39.422 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_mon_client_mode": "secure crc", 2024-08-30T15:59:39.422 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_mon_cluster_mode": "secure crc", 2024-08-30T15:59:39.422 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_mon_service_mode": "secure crc", 2024-08-30T15:59:39.422 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_osd_compress_min_size": "1024", 2024-08-30T15:59:39.422 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_osd_compress_mode": "none", 2024-08-30T15:59:39.422 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_osd_compression_algorithm": "snappy", 2024-08-30T15:59:39.422 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-08-30T15:59:39.422 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_pq_min_cost": "65536", 2024-08-30T15:59:39.422 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_public_type": "", 2024-08-30T15:59:39.422 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_service_mode": "crc secure", 2024-08-30T15:59:39.422 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_tcp_listen_backlog": "512", 2024-08-30T15:59:39.423 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_tcp_nodelay": "true", 2024-08-30T15:59:39.423 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-08-30T15:59:39.423 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_tcp_rcvbuf": "0", 2024-08-30T15:59:39.423 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_type": "async", 2024-08-30T15:59:39.423 INFO:teuthology.orchestra.run.smithi050.stdout: "no_config_file": "false", 2024-08-30T15:59:39.423 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_completion_locks_per_session": "32", 2024-08-30T15:59:39.423 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_debug_inject_relock_delay": "false", 2024-08-30T15:59:39.423 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_inflight_op_bytes": "104857600", 2024-08-30T15:59:39.423 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_inflight_ops": "1024", 2024-08-30T15:59:39.423 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_inject_no_watch_ping": "false", 2024-08-30T15:59:39.423 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-08-30T15:59:39.423 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_tick_interval": "5.000000", 2024-08-30T15:59:39.423 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_timeout": "10.000000", 2024-08-30T15:59:39.424 INFO:teuthology.orchestra.run.smithi050.stdout: "objectstore_blackhole": "false", 2024-08-30T15:59:39.424 INFO:teuthology.orchestra.run.smithi050.stdout: "openssl_engine_opts": "", 2024-08-30T15:59:39.424 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_delay_time": "5.000000", 2024-08-30T15:59:39.424 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_hist_halflife": "1000", 2024-08-30T15:59:39.424 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_max_low_ops": "2", 2024-08-30T15:59:39.424 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_max_ops": "4", 2024-08-30T15:59:39.424 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-08-30T15:59:39.424 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_quantize_effort": "0.100000", 2024-08-30T15:59:39.424 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_slop": "0.020000", 2024-08-30T15:59:39.424 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-08-30T15:59:39.424 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_allow_recovery_below_min_size": "true", 2024-08-30T15:59:39.424 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_asio_thread_count": "2", 2024-08-30T15:59:39.424 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_async_recovery_min_cost": "100", 2024-08-30T15:59:39.425 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_auto_mark_unfound_lost": "false", 2024-08-30T15:59:39.425 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backfill_retry_interval": "30.000000", 2024-08-30T15:59:39.425 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backfill_scan_max": "512", 2024-08-30T15:59:39.425 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backfill_scan_min": "64", 2024-08-30T15:59:39.425 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backoff_on_degraded": "false", 2024-08-30T15:59:39.425 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backoff_on_peering": "false", 2024-08-30T15:59:39.425 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backoff_on_unfound": "true", 2024-08-30T15:59:39.425 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_beacon_report_interval": "300", 2024-08-30T15:59:39.425 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bench_duration": "30", 2024-08-30T15:59:39.425 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-08-30T15:59:39.425 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bench_max_block_size": "67108864", 2024-08-30T15:59:39.425 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bench_small_size_max_iops": "100", 2024-08-30T15:59:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_blkin_trace_all": "false", 2024-08-30T15:59:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_blocked_scrub_grace_period": "120", 2024-08-30T15:59:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-08-30T15:59:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-08-30T15:59:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-08-30T15:59:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_check_for_log_corruption": "false", 2024-08-30T15:59:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-08-30T15:59:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-08-30T15:59:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-08-30T15:59:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-08-30T15:59:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_class_update_on_start": "true", 2024-08-30T15:59:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_client_message_cap": "256", 2024-08-30T15:59:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_client_message_size_cap": "524288000", 2024-08-30T15:59:39.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_client_op_priority": "63", 2024-08-30T15:59:39.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_client_watch_timeout": "30", 2024-08-30T15:59:39.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_command_max_records": "256", 2024-08-30T15:59:39.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_compact_on_start": "false", 2024-08-30T15:59:39.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-08-30T15:59:39.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_crush_chooseleaf_type": "0", 2024-08-30T15:59:39.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_crush_initial_weight": "-1.000000", 2024-08-30T15:59:39.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_crush_update_on_start": "true", 2024-08-30T15:59:39.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_crush_update_weight_set": "true", 2024-08-30T15:59:39.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-08-30T15:59:39.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-08-30T15:59:39.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-08-30T15:59:39.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_drop_ping_duration": "0", 2024-08-30T15:59:39.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-08-30T15:59:39.427 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_feed_pullee": "-1", 2024-08-30T15:59:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-08-30T15:59:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-08-30T15:59:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-08-30T15:59:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_misdirected_ops": "false", 2024-08-30T15:59:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_no_acting_change": "false", 2024-08-30T15:59:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_no_purge_strays": "false", 2024-08-30T15:59:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_op_order": "false", 2024-08-30T15:59:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_pg_log_writeout": "false", 2024-08-30T15:59:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_pretend_recovery_active": "false", 2024-08-30T15:59:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-08-30T15:59:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-08-30T15:59:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_shutdown": "false", 2024-08-30T15:59:39.428 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-08-30T15:59:39.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-08-30T15:59:39.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_verify_cached_snaps": "false", 2024-08-30T15:59:39.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_verify_missing_on_start": "false", 2024-08-30T15:59:39.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_verify_snaps": "false", 2024-08-30T15:59:39.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-08-30T15:59:39.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-08-30T15:59:39.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_keys": "1024", 2024-08-30T15:59:39.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-08-30T15:59:39.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-08-30T15:59:39.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-08-30T15:59:39.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_stride": "524288", 2024-08-30T15:59:39.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-08-30T15:59:39.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_default_data_pool_replay_window": "5", 2024-08-30T15:59:39.429 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_default_notify_timeout": "30", 2024-08-30T15:59:39.430 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_delete_sleep": "0.000000", 2024-08-30T15:59:39.430 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-08-30T15:59:39.430 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-08-30T15:59:39.430 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-08-30T15:59:39.430 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_discard_disconnected_ops": "true", 2024-08-30T15:59:39.430 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_enable_op_tracker": "true", 2024-08-30T15:59:39.430 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-08-30T15:59:39.430 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_extblkdev_plugins": "vdo", 2024-08-30T15:59:39.430 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-08-30T15:59:39.430 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-08-30T15:59:39.430 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_info": "true", 2024-08-30T15:59:39.430 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_shutdown": "true", 2024-08-30T15:59:39.430 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-08-30T15:59:39.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_shutdown_timeout": "15", 2024-08-30T15:59:39.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-08-30T15:59:39.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-08-30T15:59:39.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-08-30T15:59:39.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_function_tracing": "false", 2024-08-30T15:59:39.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_grace": "80", 2024-08-30T15:59:39.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_interval": "6", 2024-08-30T15:59:39.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-08-30T15:59:39.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_min_peers": "10", 2024-08-30T15:59:39.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_min_size": "2000", 2024-08-30T15:59:39.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_stale": "600", 2024-08-30T15:59:39.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-08-30T15:59:39.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_hit_set_max_size": "100000", 2024-08-30T15:59:39.432 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_hit_set_min_size": "1000", 2024-08-30T15:59:39.432 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-08-30T15:59:39.432 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-08-30T15:59:39.432 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-08-30T15:59:39.432 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-08-30T15:59:39.432 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-08-30T15:59:39.432 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_journal_flush_on_shutdown": "true", 2024-08-30T15:59:39.432 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_journal_size": "5120", 2024-08-30T15:59:39.432 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_kill_backfill_at": "0", 2024-08-30T15:59:39.432 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_loop_before_reset_tphandle": "64", 2024-08-30T15:59:39.432 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_cache_size": "50", 2024-08-30T15:59:39.432 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_dedup": "true", 2024-08-30T15:59:39.433 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_message_max": "40", 2024-08-30T15:59:39.433 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_message_max_bytes": "10485760", 2024-08-30T15:59:39.433 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_share_max_epochs": "40", 2024-08-30T15:59:39.433 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_attr_name_len": "100", 2024-08-30T15:59:39.433 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_attr_size": "0", 2024-08-30T15:59:39.433 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_backfills": "1", 2024-08-30T15:59:39.433 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_markdown_count": "5", 2024-08-30T15:59:39.433 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_markdown_period": "600", 2024-08-30T15:59:39.433 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_object_name_len": "2048", 2024-08-30T15:59:39.433 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_object_namespace_len": "256", 2024-08-30T15:59:39.433 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_object_size": "134217728", 2024-08-30T15:59:39.433 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-08-30T15:59:39.434 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_omap_entries_per_request": "1024", 2024-08-30T15:59:39.434 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_pg_blocked_by": "16", 2024-08-30T15:59:39.434 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_pg_log_entries": "10000", 2024-08-30T15:59:39.434 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-08-30T15:59:39.434 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_pgls": "1024", 2024-08-30T15:59:39.434 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_push_cost": "8388608", 2024-08-30T15:59:39.434 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_push_objects": "10", 2024-08-30T15:59:39.434 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_scrubs": "3", 2024-08-30T15:59:39.434 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-08-30T15:59:39.434 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_trimming_pgs": "2", 2024-08-30T15:59:39.434 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_write_op_reply_len": "64", 2024-08-30T15:59:39.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_write_size": "90", 2024-08-30T15:59:39.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-08-30T15:59:39.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-08-30T15:59:39.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-08-30T15:59:39.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-08-30T15:59:39.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-08-30T15:59:39.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-08-30T15:59:39.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-08-30T15:59:39.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_override_recovery_settings": "false", 2024-08-30T15:59:39.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_profile": "balanced", 2024-08-30T15:59:39.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-08-30T15:59:39.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-08-30T15:59:39.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-08-30T15:59:39.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-08-30T15:59:39.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-08-30T15:59:39.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-08-30T15:59:39.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-08-30T15:59:39.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-08-30T15:59:39.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-08-30T15:59:39.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-08-30T15:59:39.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_skip_benchmark": "false", 2024-08-30T15:59:39.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_base": "805306368", 2024-08-30T15:59:39.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_cache_min": "134217728", 2024-08-30T15:59:39.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-08-30T15:59:39.437 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-08-30T15:59:39.437 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_target": "4294967296", 2024-08-30T15:59:39.437 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_target_autotune": "false", 2024-08-30T15:59:39.437 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-08-30T15:59:39.437 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_min_pg_log_entries": "250", 2024-08-30T15:59:39.437 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_min_recovery_priority": "0", 2024-08-30T15:59:39.437 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_heartbeat_interval": "30", 2024-08-30T15:59:39.437 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-08-30T15:59:39.437 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_report_interval": "5", 2024-08-30T15:59:39.437 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_report_max_in_flight": "2", 2024-08-30T15:59:39.437 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-08-30T15:59:39.437 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_num_cache_shards": "32", 2024-08-30T15:59:39.437 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_num_op_tracker_shard": "32", 2024-08-30T15:59:39.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_numa_auto_affinity": "true", 2024-08-30T15:59:39.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_numa_node": "-1", 2024-08-30T15:59:39.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_numa_prefer_iface": "true", 2024-08-30T15:59:39.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-08-30T15:59:39.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_objecter_finishers": "1", 2024-08-30T15:59:39.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_objectstore": "bluestore", 2024-08-30T15:59:39.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_objectstore_fuse": "false", 2024-08-30T15:59:39.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_objectstore_tracing": "false", 2024-08-30T15:59:39.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_complaint_time": "30.000000", 2024-08-30T15:59:39.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_history_duration": "600", 2024-08-30T15:59:39.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_history_size": "20", 2024-08-30T15:59:39.438 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_history_slow_op_size": "20", 2024-08-30T15:59:39.439 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-08-30T15:59:39.439 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_log_threshold": "5", 2024-08-30T15:59:39.439 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_shards": "0", 2024-08-30T15:59:39.439 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_shards_hdd": "5", 2024-08-30T15:59:39.439 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_shards_ssd": "8", 2024-08-30T15:59:39.439 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_threads_per_shard": "0", 2024-08-30T15:59:39.439 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-08-30T15:59:39.439 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-08-30T15:59:39.439 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-08-30T15:59:39.439 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_pq_min_cost": "65536", 2024-08-30T15:59:39.439 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_queue": "mclock_scheduler", 2024-08-30T15:59:39.439 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_queue_cut_off": "high", 2024-08-30T15:59:39.439 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_thread_suicide_timeout": "150", 2024-08-30T15:59:39.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_thread_timeout": "150", 2024-08-30T15:59:39.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_open_classes_on_start": "true", 2024-08-30T15:59:39.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_os_flags": "0", 2024-08-30T15:59:39.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_peering_op_priority": "255", 2024-08-30T15:59:39.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_delete_cost": "1048576", 2024-08-30T15:59:39.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_delete_priority": "5", 2024-08-30T15:59:39.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-08-30T15:59:39.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-08-30T15:59:39.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_log_dups_tracked": "3000", 2024-08-30T15:59:39.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_log_trim_max": "10000", 2024-08-30T15:59:39.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_log_trim_min": "100", 2024-08-30T15:59:39.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-08-30T15:59:39.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_object_context_cache_count": "64", 2024-08-30T15:59:39.441 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_stat_report_interval_max": "500", 2024-08-30T15:59:39.441 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-08-30T15:59:39.441 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-08-30T15:59:39.441 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-08-30T15:59:39.441 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-08-30T15:59:39.441 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-08-30T15:59:39.441 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-08-30T15:59:39.441 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_crimson": "false", 2024-08-30T15:59:39.441 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_crush_rule": "-1", 2024-08-30T15:59:39.441 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_ec_fast_read": "false", 2024-08-30T15:59:39.441 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-08-30T15:59:39.441 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_bulk": "false", 2024-08-30T15:59:39.441 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-08-30T15:59:39.442 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_nodelete": "false", 2024-08-30T15:59:39.442 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-08-30T15:59:39.442 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-08-30T15:59:39.442 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flags": "0", 2024-08-30T15:59:39.442 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-08-30T15:59:39.442 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_min_size": "0", 2024-08-30T15:59:39.442 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-08-30T15:59:39.442 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_pg_num": "32", 2024-08-30T15:59:39.442 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_pgp_num": "0", 2024-08-30T15:59:39.442 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-08-30T15:59:39.442 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_size": "2", 2024-08-30T15:59:39.442 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_type": "replicated", 2024-08-30T15:59:39.443 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-08-30T15:59:39.443 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_use_gmt_hitset": "true", 2024-08-30T15:59:39.443 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_push_per_object_cost": "1000", 2024-08-30T15:59:39.443 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_read_ec_check_for_errors": "false", 2024-08-30T15:59:39.443 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recover_clone_overlap": "true", 2024-08-30T15:59:39.443 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recover_clone_overlap_limit": "10", 2024-08-30T15:59:39.443 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_cost": "20971520", 2024-08-30T15:59:39.443 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_delay_start": "0.000000", 2024-08-30T15:59:39.443 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_active": "0", 2024-08-30T15:59:39.443 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_active_hdd": "3", 2024-08-30T15:59:39.443 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_active_ssd": "10", 2024-08-30T15:59:39.443 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_chunk": "8388608", 2024-08-30T15:59:39.443 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-08-30T15:59:39.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_single_start": "1", 2024-08-30T15:59:39.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_op_priority": "3", 2024-08-30T15:59:39.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_op_warn_multiple": "16", 2024-08-30T15:59:39.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_priority": "5", 2024-08-30T15:59:39.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_retry_interval": "30.000000", 2024-08-30T15:59:39.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_sleep": "0.000000", 2024-08-30T15:59:39.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-08-30T15:59:39.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-08-30T15:59:39.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-08-30T15:59:39.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_repair_during_recovery": "false", 2024-08-30T15:59:39.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_requested_scrub_priority": "120", 2024-08-30T15:59:39.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-08-30T15:59:39.445 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_rollback_to_cluster_snap": "", 2024-08-30T15:59:39.445 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_auto_repair": "false", 2024-08-30T15:59:39.445 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-08-30T15:59:39.445 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-08-30T15:59:39.445 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_begin_hour": "0", 2024-08-30T15:59:39.445 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_begin_week_day": "0", 2024-08-30T15:59:39.445 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_chunk_max": "25", 2024-08-30T15:59:39.445 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_chunk_min": "5", 2024-08-30T15:59:39.445 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_cost": "52428800", 2024-08-30T15:59:39.445 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_during_recovery": "false", 2024-08-30T15:59:39.445 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_end_hour": "0", 2024-08-30T15:59:39.445 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_end_week_day": "0", 2024-08-30T15:59:39.446 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_event_cost": "4096", 2024-08-30T15:59:39.446 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-08-30T15:59:39.446 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-08-30T15:59:39.446 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_invalid_stats": "true", 2024-08-30T15:59:39.446 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_load_threshold": "0.500000", 2024-08-30T15:59:39.446 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_max_interval": "604800.000000", 2024-08-30T15:59:39.446 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_max_preemptions": "5", 2024-08-30T15:59:39.446 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_min_interval": "86400.000000", 2024-08-30T15:59:39.446 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_priority": "5", 2024-08-30T15:59:39.446 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_reservation_timeout": "5000", 2024-08-30T15:59:39.446 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_sleep": "0.000000", 2024-08-30T15:59:39.447 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_slow_reservation_response": "2200", 2024-08-30T15:59:39.447 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-08-30T15:59:39.447 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-08-30T15:59:39.447 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_shutdown_pgref_assert": "false", 2024-08-30T15:59:39.447 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_skip_data_digest": "false", 2024-08-30T15:59:39.447 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_smart_report_timeout": "5", 2024-08-30T15:59:39.447 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_cost": "1048576", 2024-08-30T15:59:39.447 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_priority": "5", 2024-08-30T15:59:39.447 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_sleep": "0.000000", 2024-08-30T15:59:39.447 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-08-30T15:59:39.447 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-08-30T15:59:39.447 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-08-30T15:59:39.448 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-08-30T15:59:39.448 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_stats_update_period_scrubbing": "15", 2024-08-30T15:59:39.448 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-08-30T15:59:39.448 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_target_transaction_size": "30", 2024-08-30T15:59:39.448 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-08-30T15:59:39.448 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-08-30T15:59:39.448 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-08-30T15:59:39.448 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-08-30T15:59:39.448 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-08-30T15:59:39.448 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-08-30T15:59:39.448 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-08-30T15:59:39.448 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_mode": "writeback", 2024-08-30T15:59:39.449 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-08-30T15:59:39.449 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-08-30T15:59:39.449 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tracing": "false", 2024-08-30T15:59:39.449 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_use_stale_snap": "false", 2024-08-30T15:59:39.449 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-08-30T15:59:39.449 INFO:teuthology.orchestra.run.smithi050.stdout: "osdc_blkin_trace_all": "false", 2024-08-30T15:59:39.449 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_kill_at": "0", 2024-08-30T15:59:39.449 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_max_join_drift": "10", 2024-08-30T15:59:39.449 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_min": "500", 2024-08-30T15:59:39.449 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_min_wait": "0.050000", 2024-08-30T15:59:39.449 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_propose_interval": "1.000000", 2024-08-30T15:59:39.449 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_service_trim_max": "500", 2024-08-30T15:59:39.450 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_service_trim_max_multiplier": "20", 2024-08-30T15:59:39.450 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_service_trim_min": "10", 2024-08-30T15:59:39.450 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_stash_full_interval": "25", 2024-08-30T15:59:39.450 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_trim_max": "500", 2024-08-30T15:59:39.450 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_trim_min": "250", 2024-08-30T15:59:39.450 INFO:teuthology.orchestra.run.smithi050.stdout: "perf": "true", 2024-08-30T15:59:39.450 INFO:teuthology.orchestra.run.smithi050.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-08-30T15:59:39.450 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-08-30T15:59:39.450 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-08-30T15:59:39.450 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "-", 2024-08-30T15:59:39.450 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrv": "", 2024-08-30T15:59:39.451 INFO:teuthology.orchestra.run.smithi050.stdout: "public_bind_addr": "-", 2024-08-30T15:59:39.451 INFO:teuthology.orchestra.run.smithi050.stdout: "public_network": "", 2024-08-30T15:59:39.451 INFO:teuthology.orchestra.run.smithi050.stdout: "public_network_interface": "", 2024-08-30T15:59:39.451 INFO:teuthology.orchestra.run.smithi050.stdout: "qat_compressor_enabled": "false", 2024-08-30T15:59:39.451 INFO:teuthology.orchestra.run.smithi050.stdout: "qat_compressor_session_max_number": "256", 2024-08-30T15:59:39.451 INFO:teuthology.orchestra.run.smithi050.stdout: "rados_mon_op_timeout": "0", 2024-08-30T15:59:39.451 INFO:teuthology.orchestra.run.smithi050.stdout: "rados_osd_op_timeout": "0", 2024-08-30T15:59:39.451 INFO:teuthology.orchestra.run.smithi050.stdout: "rados_tracing": "false", 2024-08-30T15:59:39.451 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_atime_update_interval": "60", 2024-08-30T15:59:39.451 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-08-30T15:59:39.451 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_balance_parent_reads": "false", 2024-08-30T15:59:39.451 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_balance_snap_reads": "false", 2024-08-30T15:59:39.452 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_blkin_trace_all": "false", 2024-08-30T15:59:39.452 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_blocklist_expire_seconds": "0", 2024-08-30T15:59:39.452 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_blocklist_on_break_lock": "true", 2024-08-30T15:59:39.452 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache": "true", 2024-08-30T15:59:39.452 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_block_writes_upfront": "false", 2024-08-30T15:59:39.452 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_max_dirty": "25165824", 2024-08-30T15:59:39.452 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-08-30T15:59:39.452 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_max_dirty_object": "0", 2024-08-30T15:59:39.452 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_policy": "writearound", 2024-08-30T15:59:39.452 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_size": "33554432", 2024-08-30T15:59:39.452 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_target_dirty": "16777216", 2024-08-30T15:59:39.452 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-08-30T15:59:39.453 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_clone_copy_on_read": "false", 2024-08-30T15:59:39.453 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_compression_hint": "none", 2024-08-30T15:59:39.453 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_concurrent_management_ops": "10", 2024-08-30T15:59:39.453 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-08-30T15:59:39.453 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_clone_format": "auto", 2024-08-30T15:59:39.453 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_data_pool": "", 2024-08-30T15:59:39.453 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_features": "61", 2024-08-30T15:59:39.453 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_format": "2", 2024-08-30T15:59:39.453 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_map_options": "", 2024-08-30T15:59:39.453 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_order": "22", 2024-08-30T15:59:39.453 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_pool": "rbd", 2024-08-30T15:59:39.453 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-08-30T15:59:39.453 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_stripe_count": "0", 2024-08-30T15:59:39.454 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_stripe_unit": "0", 2024-08-30T15:59:39.454 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_disable_zero_copy_writes": "true", 2024-08-30T15:59:39.454 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_discard_granularity_bytes": "65536", 2024-08-30T15:59:39.454 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-08-30T15:59:39.454 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_enable_alloc_hint": "true", 2024-08-30T15:59:39.454 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-08-30T15:59:39.454 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_io_scheduler": "simple", 2024-08-30T15:59:39.454 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-08-30T15:59:39.454 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_commit_age": "5.000000", 2024-08-30T15:59:39.454 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-08-30T15:59:39.454 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-08-30T15:59:39.454 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-08-30T15:59:39.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-08-30T15:59:39.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_flush_interval": "0", 2024-08-30T15:59:39.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-08-30T15:59:39.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-08-30T15:59:39.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_order": "24", 2024-08-30T15:59:39.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_pool": "", 2024-08-30T15:59:39.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_splay_width": "4", 2024-08-30T15:59:39.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_localize_parent_reads": "false", 2024-08-30T15:59:39.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_localize_snap_reads": "false", 2024-08-30T15:59:39.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-08-30T15:59:39.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-08-30T15:59:39.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-08-30T15:59:39.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-08-30T15:59:39.456 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-08-30T15:59:39.456 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-08-30T15:59:39.456 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_policy_type": "simple", 2024-08-30T15:59:39.456 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-08-30T15:59:39.456 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-08-30T15:59:39.456 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-08-30T15:59:39.456 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-08-30T15:59:39.456 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-08-30T15:59:39.456 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-08-30T15:59:39.456 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-08-30T15:59:39.456 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_autotune": "true", 2024-08-30T15:59:39.456 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_base": "805306368", 2024-08-30T15:59:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-08-30T15:59:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-08-30T15:59:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-08-30T15:59:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-08-30T15:59:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_target": "4294967296", 2024-08-30T15:59:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-08-30T15:59:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-08-30T15:59:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-08-30T15:59:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirroring_delete_delay": "0", 2024-08-30T15:59:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-08-30T15:59:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirroring_replay_delay": "0", 2024-08-30T15:59:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-08-30T15:59:39.457 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-08-30T15:59:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_move_to_trash_on_remove": "false", 2024-08-30T15:59:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-08-30T15:59:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mtime_update_interval": "60", 2024-08-30T15:59:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_non_blocking_aio": "true", 2024-08-30T15:59:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_op_thread_timeout": "60", 2024-08-30T15:59:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_op_threads": "1", 2024-08-30T15:59:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_parent_cache_enabled": "false", 2024-08-30T15:59:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_persistent_cache_mode": "disabled", 2024-08-30T15:59:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_persistent_cache_path": "/tmp", 2024-08-30T15:59:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_persistent_cache_size": "1073741824", 2024-08-30T15:59:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_plugins": "", 2024-08-30T15:59:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_bps_burst": "0", 2024-08-30T15:59:39.458 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-08-30T15:59:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_bps_limit": "0", 2024-08-30T15:59:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_exclude_ops": "0", 2024-08-30T15:59:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_iops_burst": "0", 2024-08-30T15:59:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-08-30T15:59:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_iops_limit": "0", 2024-08-30T15:59:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_bps_burst": "0", 2024-08-30T15:59:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-08-30T15:59:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_bps_limit": "0", 2024-08-30T15:59:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_iops_burst": "0", 2024-08-30T15:59:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-08-30T15:59:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_iops_limit": "0", 2024-08-30T15:59:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_schedule_tick_min": "50", 2024-08-30T15:59:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_bps_burst": "0", 2024-08-30T15:59:39.459 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-08-30T15:59:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_bps_limit": "0", 2024-08-30T15:59:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_iops_burst": "0", 2024-08-30T15:59:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-08-30T15:59:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_iops_limit": "0", 2024-08-30T15:59:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_quiesce_notification_attempts": "10", 2024-08-30T15:59:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_read_from_replica_policy": "default", 2024-08-30T15:59:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-08-30T15:59:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_readahead_max_bytes": "524288", 2024-08-30T15:59:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_readahead_trigger_requests": "10", 2024-08-30T15:59:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_request_timed_out_seconds": "30", 2024-08-30T15:59:39.460 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_skip_partial_discard": "true", 2024-08-30T15:59:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-08-30T15:59:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_tracing": "false", 2024-08-30T15:59:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_validate_names": "true", 2024-08-30T15:59:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_validate_pool": "true", 2024-08-30T15:59:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "restapi_base_url": "", 2024-08-30T15:59:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "restapi_log_level": "", 2024-08-30T15:59:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_acl_grants_max_num": "100", 2024-08-30T15:59:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_admin_entry": "admin", 2024-08-30T15:59:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-08-30T15:59:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_backend_store": "rados", 2024-08-30T15:59:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_barbican_url": "", 2024-08-30T15:59:39.461 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_beast_enable_async": "true", 2024-08-30T15:59:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-08-30T15:59:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-08-30T15:59:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_index_max_aio": "128", 2024-08-30T15:59:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-08-30T15:59:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-08-30T15:59:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_quota_ttl": "600", 2024-08-30T15:59:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-08-30T15:59:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cache_enabled": "true", 2024-08-30T15:59:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cache_expiry_interval": "900", 2024-08-30T15:59:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cache_lru_size": "10000", 2024-08-30T15:59:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_config_store": "rados", 2024-08-30T15:59:39.462 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_content_length_compat": "false", 2024-08-30T15:59:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_copy_obj_progress": "true", 2024-08-30T15:59:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-08-30T15:59:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cors_rules_max_num": "100", 2024-08-30T15:59:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cross_domain_policy": "", 2024-08-30T15:59:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_default_encryption_key": "", 2024-08-30T15:59:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_addr": "", 2024-08-30T15:59:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_ca_path": "", 2024-08-30T15:59:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_client_cert": "", 2024-08-30T15:59:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_client_key": "", 2024-08-30T15:59:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-08-30T15:59:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_password": "", 2024-08-30T15:59:39.463 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-08-30T15:59:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_username": "", 2024-08-30T15:59:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_require_ssl": "true", 2024-08-30T15:59:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-08-30T15:59:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-08-30T15:59:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-08-30T15:59:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-08-30T15:59:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-08-30T15:59:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-08-30T15:59:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-08-30T15:59:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-08-30T15:59:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-08-30T15:59:39.464 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-08-30T15:59:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-08-30T15:59:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-08-30T15:59:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-08-30T15:59:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-08-30T15:59:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_suppress_logs": "true", 2024-08-30T15:59:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_addr": "", 2024-08-30T15:59:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_auth": "token", 2024-08-30T15:59:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_namespace": "", 2024-08-30T15:59:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_prefix": "", 2024-08-30T15:59:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-08-30T15:59:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-08-30T15:59:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-08-30T15:59:39.465 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-08-30T15:59:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_token_file": "", 2024-08-30T15:59:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-08-30T15:59:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_buffersize": "524288", 2024-08-30T15:59:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_low_speed_limit": "1024", 2024-08-30T15:59:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_low_speed_time": "300", 2024-08-30T15:59:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_tcp_keepalive": "0", 2024-08-30T15:59:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-08-30T15:59:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-08-30T15:59:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-08-30T15:59:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-08-30T15:59:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-08-30T15:59:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_fadvise": "4", 2024-08-30T15:59:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-08-30T15:59:39.466 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-08-30T15:59:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-08-30T15:59:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-08-30T15:59:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_log_changes_size": "1000", 2024-08-30T15:59:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_log_num_shards": "128", 2024-08-30T15:59:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-08-30T15:59:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_log_window": "30", 2024-08-30T15:59:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_notify_interval_msec": "0", 2024-08-30T15:59:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_sync_poll_interval": "20", 2024-08-30T15:59:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_sync_spawn_window": "20", 2024-08-30T15:59:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-08-30T15:59:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-08-30T15:59:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_data_log_backing": "fifo", 2024-08-30T15:59:39.467 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-08-30T15:59:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_region_info_oid": "default.region", 2024-08-30T15:59:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-08-30T15:59:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-08-30T15:59:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_defer_to_bucket_acls": "", 2024-08-30T15:59:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-08-30T15:59:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-08-30T15:59:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-08-30T15:59:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-08-30T15:59:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-08-30T15:59:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-08-30T15:59:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-08-30T15:59:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-08-30T15:59:39.468 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_data_res": "500.000000", 2024-08-30T15:59:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-08-30T15:59:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-08-30T15:59:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-08-30T15:59:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-08-30T15:59:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dns_name": "", 2024-08-30T15:59:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dns_s3website_name": "", 2024-08-30T15:59:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dynamic_resharding": "true", 2024-08-30T15:59:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-08-30T15:59:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_gc_threads": "true", 2024-08-30T15:59:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_lc_threads": "true", 2024-08-30T15:59:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_ops_log": "false", 2024-08-30T15:59:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_quota_threads": "true", 2024-08-30T15:59:39.469 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_static_website": "false", 2024-08-30T15:59:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_usage_log": "false", 2024-08-30T15:59:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enforce_swift_acls": "true", 2024-08-30T15:59:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_exit_timeout_secs": "120", 2024-08-30T15:59:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_expose_bucket": "false", 2024-08-30T15:59:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_extended_http_attrs": "", 2024-08-30T15:59:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_filter": "none", 2024-08-30T15:59:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-08-30T15:59:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_frontends": "beast port=7480", 2024-08-30T15:59:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_concurrent_io": "10", 2024-08-30T15:59:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_deferred": "50", 2024-08-30T15:59:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-08-30T15:59:39.470 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_objs": "32", 2024-08-30T15:59:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_queue_size": "134213632", 2024-08-30T15:59:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_trim_chunk": "16", 2024-08-30T15:59:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_obj_min_wait": "7200", 2024-08-30T15:59:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_processor_max_time": "3600", 2024-08-30T15:59:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_processor_period": "3600", 2024-08-30T15:59:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-08-30T15:59:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_get_obj_window_size": "16777216", 2024-08-30T15:59:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_healthcheck_disabling_path": "", 2024-08-30T15:59:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ignore_get_invalid_range": "false", 2024-08-30T15:59:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_init_timeout": "300", 2024-08-30T15:59:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-08-30T15:59:39.471 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-08-30T15:59:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-08-30T15:59:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-08-30T15:59:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_domain": "", 2024-08-30T15:59:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_password": "", 2024-08-30T15:59:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_password_path": "", 2024-08-30T15:59:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_project": "", 2024-08-30T15:59:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_tenant": "", 2024-08-30T15:59:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_token": "", 2024-08-30T15:59:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_token_path": "", 2024-08-30T15:59:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_user": "", 2024-08-30T15:59:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_api_version": "2", 2024-08-30T15:59:39.472 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_domain": "", 2024-08-30T15:59:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_password": "", 2024-08-30T15:59:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_project": "", 2024-08-30T15:59:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_tenant": "", 2024-08-30T15:59:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_user": "", 2024-08-30T15:59:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-08-30T15:59:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_implicit_tenants": "false", 2024-08-30T15:59:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-08-30T15:59:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_service_token_enabled": "false", 2024-08-30T15:59:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_token_cache_size": "10000", 2024-08-30T15:59:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_url": "", 2024-08-30T15:59:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_verify_ssl": "true", 2024-08-30T15:59:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_debug_interval": "-1", 2024-08-30T15:59:39.473 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_lock_max_time": "90", 2024-08-30T15:59:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_max_objs": "32", 2024-08-30T15:59:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_max_rules": "1000", 2024-08-30T15:59:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_max_worker": "3", 2024-08-30T15:59:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_max_wp_worker": "3", 2024-08-30T15:59:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_thread_delay": "0", 2024-08-30T15:59:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-08-30T15:59:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_dnattr": "uid", 2024-08-30T15:59:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-08-30T15:59:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_searchfilter": "", 2024-08-30T15:59:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-08-30T15:59:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_uri": "ldaps://", 2024-08-30T15:59:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-08-30T15:59:39.474 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-08-30T15:59:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-08-30T15:59:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_log_http_headers": "", 2024-08-30T15:59:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_log_nonexistent_bucket": "false", 2024-08-30T15:59:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-08-30T15:59:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_log_object_name_utc": "false", 2024-08-30T15:59:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-08-30T15:59:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_attr_name_len": "0", 2024-08-30T15:59:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_attr_size": "0", 2024-08-30T15:59:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_attrs_num_in_req": "0", 2024-08-30T15:59:39.475 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_chunk_size": "4194304", 2024-08-30T15:59:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_concurrent_requests": "1024", 2024-08-30T15:59:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-08-30T15:59:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_dynamic_shards": "1999", 2024-08-30T15:59:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_listing_results": "1000", 2024-08-30T15:59:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_notify_retries": "10", 2024-08-30T15:59:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_objs_per_shard": "100000", 2024-08-30T15:59:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_put_param_size": "1048576", 2024-08-30T15:59:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_put_size": "5368709120", 2024-08-30T15:59:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_slo_entries": "1000", 2024-08-30T15:59:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_md_log_max_shards": "64", 2024-08-30T15:59:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_md_notify_interval_msec": "200", 2024-08-30T15:59:39.476 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_meta_sync_poll_interval": "20", 2024-08-30T15:59:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_meta_sync_spawn_window": "20", 2024-08-30T15:59:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-08-30T15:59:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_mp_lock_max_time": "600", 2024-08-30T15:59:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-08-30T15:59:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_multipart_min_part_size": "5242880", 2024-08-30T15:59:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-08-30T15:59:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-08-30T15:59:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_fhcache_size": "2017", 2024-08-30T15:59:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-08-30T15:59:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-08-30T15:59:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_lru_lanes": "5", 2024-08-30T15:59:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_max_gc": "300", 2024-08-30T15:59:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-08-30T15:59:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_run_gc_threads": "false", 2024-08-30T15:59:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_run_lc_threads": "false", 2024-08-30T15:59:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_run_quota_threads": "false", 2024-08-30T15:59:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_run_sync_thread": "false", 2024-08-30T15:59:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-08-30T15:59:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-08-30T15:59:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_num_async_rados_threads": "32", 2024-08-30T15:59:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_num_control_oids": "8", 2024-08-30T15:59:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_numa_node": "-1", 2024-08-30T15:59:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_obj_stripe_size": "4194304", 2024-08-30T15:59:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-08-30T15:59:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_objexp_chunk_size": "100", 2024-08-30T15:59:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_objexp_gc_interval": "600", 2024-08-30T15:59:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_objexp_hints_num_shards": "127", 2024-08-30T15:59:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-08-30T15:59:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-08-30T15:59:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_op_thread_timeout": "600", 2024-08-30T15:59:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_op_tracing": "false", 2024-08-30T15:59:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_opa_token": "", 2024-08-30T15:59:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_opa_url": "", 2024-08-30T15:59:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_opa_verify_ssl": "true", 2024-08-30T15:59:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-08-30T15:59:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-08-30T15:59:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ops_log_rados": "false", 2024-08-30T15:59:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ops_log_socket_path": "", 2024-08-30T15:59:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-08-30T15:59:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-08-30T15:59:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-08-30T15:59:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_period_push_interval": "2.000000", 2024-08-30T15:59:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_period_push_interval_max": "30.000000", 2024-08-30T15:59:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_period_root_pool": ".rgw.root", 2024-08-30T15:59:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-08-30T15:59:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_print_continue": "true", 2024-08-30T15:59:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_print_prohibited_content_length": "false", 2024-08-30T15:59:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-08-30T15:59:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-08-30T15:59:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-08-30T15:59:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-08-30T15:59:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_rados_tracing": "false", 2024-08-30T15:59:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_realm": "", 2024-08-30T15:59:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_realm_id": "", 2024-08-30T15:59:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-08-30T15:59:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_region": "", 2024-08-30T15:59:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_region_root_pool": ".rgw.root", 2024-08-30T15:59:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_relaxed_region_enforcement": "false", 2024-08-30T15:59:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-08-30T15:59:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-08-30T15:59:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_request_uri": "", 2024-08-30T15:59:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_batch_size": "64", 2024-08-30T15:59:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-08-30T15:59:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_max_aio": "128", 2024-08-30T15:59:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_num_logs": "16", 2024-08-30T15:59:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_thread_interval": "600", 2024-08-30T15:59:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_resolve_cname": "false", 2024-08-30T15:59:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_rest_getusage_op_compat": "false", 2024-08-30T15:59:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_run_sync_thread": "true", 2024-08-30T15:59:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-08-30T15:59:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_use_keystone": "false", 2024-08-30T15:59:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_use_ldap": "false", 2024-08-30T15:59:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_use_rados": "true", 2024-08-30T15:59:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_use_sts": "false", 2024-08-30T15:59:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-08-30T15:59:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_success_create_obj_status": "0", 2024-08-30T15:59:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-08-30T15:59:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_scheduler_type": "throttler", 2024-08-30T15:59:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_script_uri": "", 2024-08-30T15:59:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_service_provider_name": "", 2024-08-30T15:59:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-08-30T15:59:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_client_id": "", 2024-08-30T15:59:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_client_secret": "", 2024-08-30T15:59:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_entry": "sts", 2024-08-30T15:59:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_key": "sts", 2024-08-30T15:59:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_max_session_duration": "43200", 2024-08-30T15:59:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_min_session_duration": "900", 2024-08-30T15:59:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_token_introspection_url": "", 2024-08-30T15:59:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_account_in_url": "false", 2024-08-30T15:59:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_auth_entry": "auth", 2024-08-30T15:59:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_auth_url": "", 2024-08-30T15:59:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_custom_header": "", 2024-08-30T15:59:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_enforce_content_length": "false", 2024-08-30T15:59:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_need_stats": "true", 2024-08-30T15:59:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_tenant_name": "", 2024-08-30T15:59:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_token_expiration": "86400", 2024-08-30T15:59:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_url": "", 2024-08-30T15:59:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_url_prefix": "swift", 2024-08-30T15:59:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_versioning_enabled": "false", 2024-08-30T15:59:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-08-30T15:59:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-08-30T15:59:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_lease_period": "120", 2024-08-30T15:59:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-08-30T15:59:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_log_trim_interval": "1200", 2024-08-30T15:59:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-08-30T15:59:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-08-30T15:59:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-08-30T15:59:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_obj_etag_verify": "false", 2024-08-30T15:59:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_trace_history_size": "4096", 2024-08-30T15:59:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-08-30T15:59:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-08-30T15:59:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_thread_pool_size": "512", 2024-08-30T15:59:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_comment": "", 2024-08-30T15:59:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_createby": "", 2024-08-30T15:59:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_encoding": "", 2024-08-30T15:59:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_flag": "false", 2024-08-30T15:59:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_origin": "", 2024-08-30T15:59:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_sha_unit": "524288", 2024-08-30T15:59:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_tracker": "", 2024-08-30T15:59:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_trust_forwarded_https": "false", 2024-08-30T15:59:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-08-30T15:59:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_usage_log_tick_interval": "30", 2024-08-30T15:59:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_usage_max_shards": "32", 2024-08-30T15:59:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_usage_max_user_shards": "1", 2024-08-30T15:59:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_use_opa_authz": "false", 2024-08-30T15:59:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-08-30T15:59:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_default_quota_max_size": "-1", 2024-08-30T15:59:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_max_buckets": "1000", 2024-08-30T15:59:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_policies_max_num": "100", 2024-08-30T15:59:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-08-30T15:59:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-08-30T15:59:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_quota_sync_interval": "86400", 2024-08-30T15:59:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-08-30T15:59:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_unique_email": "true", 2024-08-30T15:59:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_verify_ssl": "true", 2024-08-30T15:59:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_website_routing_rules_max_num": "50", 2024-08-30T15:59:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zone": "", 2024-08-30T15:59:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zone_id": "", 2024-08-30T15:59:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-08-30T15:59:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zonegroup": "", 2024-08-30T15:59:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zonegroup_id": "", 2024-08-30T15:59:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-08-30T15:59:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "rgwlc_auto_session_clear": "true", 2024-08-30T15:59:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "rgwlc_skip_bucket_step": "false", 2024-08-30T15:59:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_block_size": "4096", 2024-08-30T15:59:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-08-30T15:59:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-08-30T15:59:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-08-30T15:59:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-08-30T15:59:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_shard_bits": "4", 2024-08-30T15:59:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_size": "536870912", 2024-08-30T15:59:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_type": "binned_lru", 2024-08-30T15:59:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-08-30T15:59:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-08-30T15:59:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-08-30T15:59:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_collect_compaction_stats": "false", 2024-08-30T15:59:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_collect_extended_stats": "false", 2024-08-30T15:59:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_collect_memory_stats": "false", 2024-08-30T15:59:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-08-30T15:59:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_index_type": "binary_search", 2024-08-30T15:59:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_log_to_ceph_log": "true", 2024-08-30T15:59:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_metadata_block_size": "4096", 2024-08-30T15:59:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_partition_filters": "false", 2024-08-30T15:59:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_perf": "false", 2024-08-30T15:59:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-08-30T15:59:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-08-30T15:59:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "rotating_keys_renewal_timeout": "10", 2024-08-30T15:59:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "run_dir": "/var/run/ceph", 2024-08-30T15:59:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_block_create": "true", 2024-08-30T15:59:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_cache_lru_size": "67108864", 2024-08-30T15:59:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_cbjournal_size": "5368709120", 2024-08-30T15:59:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_default_max_object_size": "16777216", 2024-08-30T15:59:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-08-30T15:59:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_device_size": "53687091200", 2024-08-30T15:59:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_journal_batch_capacity": "16", 2024-08-30T15:59:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-08-30T15:59:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-08-30T15:59:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_journal_iodepth_limit": "5", 2024-08-30T15:59:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_main_device_type": "SSD", 2024-08-30T15:59:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_max_concurrent_transactions": "8", 2024-08-30T15:59:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-08-30T15:59:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-08-30T15:59:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-08-30T15:59:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-08-30T15:59:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_segment_size": "67108864", 2024-08-30T15:59:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "set_keepcaps": "false", 2024-08-30T15:59:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "setgroup": "", 2024-08-30T15:59:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "setuser": "", 2024-08-30T15:59:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "setuser_match_path": "", 2024-08-30T15:59:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_misplaced_ratio": "0.050000", 2024-08-30T15:59:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "thp": "false", 2024-08-30T15:59:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "threadpool_default_timeout": "60", 2024-08-30T15:59:39.494 INFO:teuthology.orchestra.run.smithi050.stdout: "threadpool_empty_queue_max_wait": "2", 2024-08-30T15:59:39.494 INFO:teuthology.orchestra.run.smithi050.stdout: "throttler_perf_counter": "true" 2024-08-30T15:59:39.494 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:39.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-08-30T15:59:39.749 INFO:teuthology.orchestra.run.smithi050.stdout:mon.a: {} 2024-08-30T15:59:39.749 INFO:teuthology.orchestra.run.smithi050.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-30T15:59:39.850 INFO:teuthology.orchestra.run.smithi050.stdout:mon.b: {} 2024-08-30T15:59:39.851 INFO:teuthology.orchestra.run.smithi050.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-30T15:59:39.969 INFO:teuthology.orchestra.run.smithi050.stdout:mon.c: {} 2024-08-30T15:59:39.969 INFO:teuthology.orchestra.run.smithi050.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-30T15:59:39.979 DEBUG:teuthology.orchestra.run.smithi050:> type ceph-objectstore-tool 2024-08-30T15:59:39.994 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-08-30T15:59:39.994 DEBUG:teuthology.orchestra.run.smithi094:> type ceph-objectstore-tool 2024-08-30T15:59:40.009 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-08-30T15:59:40.009 INFO:teuthology.run_tasks:Running task mon_recovery... 2024-08-30T15:59:40.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-30T15:59:40.023 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-08-30T15:59:40.023 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T15:59:40.024 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T15:59:40.024 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-08-30T15:59:40.024 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-08-30T15:59:40.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T15:59:40.025 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-08-30T15:59:40.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T15:59:40.025 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-08-30T15:59:40.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-08-30T15:59:40.056 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:40.053+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:40.132 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:40.128+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:40.184 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:40.185 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T15:59:40.185 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T15:59:40.185 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:40.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T15:59:40.246 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:40.242+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:40.328 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:40.326+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:40.334 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T15:59:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T15:59:40.389 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:40.389 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:40.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T15:59:40.445 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:40.440+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:40.491 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T15:59:40.491 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T15:59:40.530 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:40.528+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:40.591 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:40.592 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:40.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T15:59:40.716 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T15:59:40.716 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T15:59:40.721 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:40.715+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:40.731 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T15:59:40.731 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":29,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T15:59:33.399292+0000","last_up_change":"2024-08-30T15:59:19.280372+0000","last_in_change":"2024-08-30T15:58:25.430425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-30T15:59:08.355540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-30T15:59:21.696796+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6812","nonce":1957399494}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6813","nonce":1957399494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6815","nonce":1957399494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6814","nonce":1957399494}]},"public_addr":"172.21.15.50:6812/1957399494","cluster_addr":"172.21.15.50:6813/1957399494","heartbeat_back_addr":"172.21.15.50:6815/1957399494","heartbeat_front_addr":"172.21.15.50:6814/1957399494","state":["exists","up"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6808","nonce":3208178756}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6809","nonce":3208178756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6811","nonce":3208178756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6810","nonce":3208178756}]},"public_addr":"172.21.15.50:6808/3208178756","cluster_addr":"172.21.15.50:6809/3208178756","heartbeat_back_addr":"172.21.15.50:6811/3208178756","heartbeat_front_addr":"172.21.15.50:6810/3208178756","state":["exists","up"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6804","nonce":287018800}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6805","nonce":287018800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6807","nonce":287018800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6806","nonce":287018800}]},"public_addr":"172.21.15.50:6804/287018800","cluster_addr":"172.21.15.50:6805/287018800","heartbeat_back_addr":"172.21.15.50:6807/287018800","heartbeat_front_addr":"172.21.15.50:6806/287018800","state":["exists","up"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6800","nonce":1140717714}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6801","nonce":1140717714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6803","nonce":1140717714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6802","nonce":1140717714}]},"public_addr":"172.21.15.50:6800/1140717714","cluster_addr":"172.21.15.50:6801/1140717714","heartbeat_back_addr":"172.21.15.50:6803/1140717714","heartbeat_front_addr":"172.21.15.50:6802/1140717714","state":["exists","up"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":1468073345}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":1468073345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":1468073345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":1468073345}]},"public_addr":"172.21.15.94:6805/1468073345","cluster_addr":"172.21.15.94:6806/1468073345","heartbeat_back_addr":"172.21.15.94:6808/1468073345","heartbeat_front_addr":"172.21.15.94:6807/1468073345","state":["exists","up"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":3876025616}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":3876025616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":3876025616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":3876025616}]},"public_addr":"172.21.15.94:6809/3876025616","cluster_addr":"172.21.15.94:6810/3876025616","heartbeat_back_addr":"172.21.15.94:6812/3876025616","heartbeat_front_addr":"172.21.15.94:6811/3876025616","state":["exists","up"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":128519122}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":128519122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":128519122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":128519122}]},"public_addr":"172.21.15.94:6801/128519122","cluster_addr":"172.21.15.94:6802/128519122","heartbeat_back_addr":"172.21.15.94:6804/128519122","heartbeat_front_addr":"172.21.15.94:6803/128519122","state":["exists","up"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":68721277}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":68721277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6816","nonce":68721277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":68721277}]},"public_addr":"172.21.15.94:6813/68721277","cluster_addr":"172.21.15.94:6814/68721277","heartbeat_back_addr":"172.21.15.94:6816/68721277","heartbeat_front_addr":"172.21.15.94:6815/68721277","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.911895+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.786543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.197952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.680847+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:01.937600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:02.082761+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.401025+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.129371+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T15:59:40.732 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:40.730+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:40.749 INFO:tasks.mon_recovery.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T15:59:08.355540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-08-30T15:59:21.696796+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-30T15:59:40.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-08-30T15:59:40.812 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:40.812 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:40.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T15:59:40.837 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:40.835+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:40.843 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T15:59:40.843 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T15:59:40.933 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:40.932+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:40.956 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:40.957 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:40.963 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T15:59:40.963 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T15:59:40.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T15:59:41.040 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:41.037+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:41.083 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T15:59:41.083 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T15:59:41.136 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:41.137 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:41.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T15:59:41.193 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T15:59:41.194 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T15:59:41.194 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:41.193+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:41.236 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:41.234+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:41.283 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:41.283 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:41.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T15:59:41.302 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T15:59:41.303 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T15:59:41.334 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-08-30T15:59:41.337 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:41.335+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:41.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-08-30T15:59:41.433 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:41.433 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:41.438 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:41.436+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:41.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T15:59:41.538 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:41.538+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:41.628 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:41.628 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:41.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T15:59:41.640 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:41.637+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:41.740 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:41.737+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:41.773 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:41.773 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:41.784 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T15:59:41.835 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-08-30T15:59:41.849 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:41.843+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:41.950 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:41.948+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:41.966 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:41.966 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:41.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T15:59:41.988 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 8 2024-08-30T15:59:42.001 INFO:tasks.mon_recovery:mon ids = ['a', 'c', 'b'] 2024-08-30T15:59:42.001 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-08-30T15:59:42.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T15:59:42.043 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:42.042+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:42.121 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:42.121 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:42.131 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T15:59:42.143 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:42.141+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:42.244 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:42.244+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:42.271 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:42.271 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:42.280 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T15:59:42.364 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:42.363+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:42.445 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:42.445+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:42.455 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:42.455 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:42.465 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T15:59:42.546 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:42.544+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:42.629 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:42.629 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:42.638 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T15:59:42.682 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:42.681+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:42.699 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":82,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T15:59:42.713 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-08-30T15:59:42.713 INFO:tasks.mon_recovery:verifying all monitors are in the quorum 2024-08-30T15:59:42.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-30T15:59:42.748 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:42.747+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:42.772 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:42.772 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:42.781 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T15:59:42.849 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:42.846+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:42.950 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:42.949+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:42.956 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:42.956 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:42.965 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T15:59:43.051 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:43.050+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:43.061 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:43.061 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2024-08-30T15:59:43.061 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2024-08-30T15:59:43.061 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "leader", 2024-08-30T15:59:43.062 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 6, 2024-08-30T15:59:43.062 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2024-08-30T15:59:43.062 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2024-08-30T15:59:43.062 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2024-08-30T15:59:43.062 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2024-08-30T15:59:43.062 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.062 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 83, 2024-08-30T15:59:43.062 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2024-08-30T15:59:43.062 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2024-08-30T15:59:43.062 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2024-08-30T15:59:43.062 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2024-08-30T15:59:43.062 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2024-08-30T15:59:43.063 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2024-08-30T15:59:43.063 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2024-08-30T15:59:43.063 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2024-08-30T15:59:43.063 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2024-08-30T15:59:43.063 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2024-08-30T15:59:43.063 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2024-08-30T15:59:43.063 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2024-08-30T15:59:43.063 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2024-08-30T15:59:43.063 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.063 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2024-08-30T15:59:43.063 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2024-08-30T15:59:43.063 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2024-08-30T15:59:43.063 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2024-08-30T15:59:43.064 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2024-08-30T15:59:43.064 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2024-08-30T15:59:43.064 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2024-08-30T15:59:43.064 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2024-08-30T15:59:43.064 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2024-08-30T15:59:43.064 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2024-08-30T15:59:43.064 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2024-08-30T15:59:43.064 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2024-08-30T15:59:43.064 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.064 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2024-08-30T15:59:43.064 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2024-08-30T15:59:43.065 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2024-08-30T15:59:43.065 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2024-08-30T15:59:43.065 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2024-08-30T15:59:43.065 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "9d72e2e2-0554-42af-8e62-9b1a3a804112", 2024-08-30T15:59:43.065 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-08-30T15:57:29.788167Z", 2024-08-30T15:59:43.065 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-08-30T15:57:29.788167Z", 2024-08-30T15:59:43.065 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2024-08-30T15:59:43.065 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2024-08-30T15:59:43.065 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2024-08-30T15:59:43.065 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2024-08-30T15:59:43.065 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2024-08-30T15:59:43.065 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2024-08-30T15:59:43.065 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2024-08-30T15:59:43.066 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2024-08-30T15:59:43.066 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2024-08-30T15:59:43.066 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2024-08-30T15:59:43.066 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2024-08-30T15:59:43.066 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2024-08-30T15:59:43.066 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2024-08-30T15:59:43.066 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2024-08-30T15:59:43.066 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2024-08-30T15:59:43.066 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2024-08-30T15:59:43.066 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2024-08-30T15:59:43.066 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2024-08-30T15:59:43.066 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2024-08-30T15:59:43.066 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.067 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2024-08-30T15:59:43.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.067 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2024-08-30T15:59:43.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2024-08-30T15:59:43.067 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2024-08-30T15:59:43.067 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T15:59:43.067 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T15:59:43.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.067 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T15:59:43.067 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2024-08-30T15:59:43.067 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2024-08-30T15:59:43.067 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.068 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.068 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2024-08-30T15:59:43.068 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2024-08-30T15:59:43.068 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2024-08-30T15:59:43.068 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T15:59:43.068 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2024-08-30T15:59:43.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2024-08-30T15:59:43.068 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2024-08-30T15:59:43.068 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T15:59:43.068 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T15:59:43.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.069 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T15:59:43.069 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6789", 2024-08-30T15:59:43.069 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2024-08-30T15:59:43.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.069 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.069 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6789/0", 2024-08-30T15:59:43.069 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6789/0", 2024-08-30T15:59:43.069 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2024-08-30T15:59:43.069 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T15:59:43.069 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2024-08-30T15:59:43.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2024-08-30T15:59:43.070 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2024-08-30T15:59:43.070 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T15:59:43.070 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T15:59:43.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.070 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T15:59:43.070 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2024-08-30T15:59:43.070 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2024-08-30T15:59:43.070 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.070 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.070 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2024-08-30T15:59:43.071 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2024-08-30T15:59:43.071 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2024-08-30T15:59:43.071 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T15:59:43.071 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2024-08-30T15:59:43.071 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.071 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.071 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2024-08-30T15:59:43.071 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2024-08-30T15:59:43.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.071 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2024-08-30T15:59:43.071 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "reef", 2024-08-30T15:59:43.071 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2024-08-30T15:59:43.072 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.072 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.072 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2024-08-30T15:59:43.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.072 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2024-08-30T15:59:43.072 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "reef", 2024-08-30T15:59:43.072 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 2 2024-08-30T15:59:43.072 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.072 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.072 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2024-08-30T15:59:43.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.072 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2024-08-30T15:59:43.073 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "reef", 2024-08-30T15:59:43.073 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2024-08-30T15:59:43.073 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.073 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.073 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2024-08-30T15:59:43.073 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:43.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-30T15:59:43.133 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:43.133 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:43.143 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T15:59:43.185 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:43.184+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:43.253 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:43.250+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:43.282 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:43.282 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2024-08-30T15:59:43.282 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2024-08-30T15:59:43.282 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2024-08-30T15:59:43.282 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 6, 2024-08-30T15:59:43.282 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2024-08-30T15:59:43.282 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2024-08-30T15:59:43.282 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2024-08-30T15:59:43.282 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2024-08-30T15:59:43.282 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.283 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 83, 2024-08-30T15:59:43.283 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2024-08-30T15:59:43.283 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2024-08-30T15:59:43.283 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2024-08-30T15:59:43.283 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2024-08-30T15:59:43.283 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2024-08-30T15:59:43.283 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2024-08-30T15:59:43.283 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2024-08-30T15:59:43.283 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2024-08-30T15:59:43.283 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2024-08-30T15:59:43.283 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2024-08-30T15:59:43.283 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2024-08-30T15:59:43.283 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2024-08-30T15:59:43.283 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2024-08-30T15:59:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2024-08-30T15:59:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2024-08-30T15:59:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2024-08-30T15:59:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2024-08-30T15:59:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2024-08-30T15:59:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2024-08-30T15:59:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2024-08-30T15:59:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2024-08-30T15:59:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2024-08-30T15:59:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2024-08-30T15:59:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2024-08-30T15:59:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2024-08-30T15:59:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2024-08-30T15:59:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2024-08-30T15:59:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2024-08-30T15:59:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2024-08-30T15:59:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2024-08-30T15:59:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "9d72e2e2-0554-42af-8e62-9b1a3a804112", 2024-08-30T15:59:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-08-30T15:57:29.788167Z", 2024-08-30T15:59:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-08-30T15:57:29.788167Z", 2024-08-30T15:59:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2024-08-30T15:59:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2024-08-30T15:59:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2024-08-30T15:59:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2024-08-30T15:59:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2024-08-30T15:59:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2024-08-30T15:59:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2024-08-30T15:59:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2024-08-30T15:59:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2024-08-30T15:59:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2024-08-30T15:59:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2024-08-30T15:59:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2024-08-30T15:59:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2024-08-30T15:59:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2024-08-30T15:59:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2024-08-30T15:59:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2024-08-30T15:59:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2024-08-30T15:59:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2024-08-30T15:59:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2024-08-30T15:59:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2024-08-30T15:59:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2024-08-30T15:59:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2024-08-30T15:59:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2024-08-30T15:59:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T15:59:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T15:59:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T15:59:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2024-08-30T15:59:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2024-08-30T15:59:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2024-08-30T15:59:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2024-08-30T15:59:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2024-08-30T15:59:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T15:59:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2024-08-30T15:59:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2024-08-30T15:59:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2024-08-30T15:59:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T15:59:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T15:59:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T15:59:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6789", 2024-08-30T15:59:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2024-08-30T15:59:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6789/0", 2024-08-30T15:59:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6789/0", 2024-08-30T15:59:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2024-08-30T15:59:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T15:59:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2024-08-30T15:59:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2024-08-30T15:59:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2024-08-30T15:59:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T15:59:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T15:59:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T15:59:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2024-08-30T15:59:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2024-08-30T15:59:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.290 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.291 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2024-08-30T15:59:43.291 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2024-08-30T15:59:43.291 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2024-08-30T15:59:43.291 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T15:59:43.291 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2024-08-30T15:59:43.291 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.291 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.291 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2024-08-30T15:59:43.291 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2024-08-30T15:59:43.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.291 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2024-08-30T15:59:43.291 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "reef", 2024-08-30T15:59:43.292 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2024-08-30T15:59:43.292 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.292 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.292 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2024-08-30T15:59:43.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.292 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2024-08-30T15:59:43.292 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "reef", 2024-08-30T15:59:43.292 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2024-08-30T15:59:43.292 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.292 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.292 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2024-08-30T15:59:43.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.292 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2024-08-30T15:59:43.293 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "reef", 2024-08-30T15:59:43.293 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2024-08-30T15:59:43.293 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.293 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.293 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2024-08-30T15:59:43.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.293 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2024-08-30T15:59:43.293 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "reef", 2024-08-30T15:59:43.293 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2024-08-30T15:59:43.293 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.293 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.293 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2024-08-30T15:59:43.293 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:43.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-30T15:59:43.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T15:59:43.321 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:43.321 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:43.331 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T15:59:43.375 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:43.374+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:43.455 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:43.452+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:43.456 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:43.456 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:43.465 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T15:59:43.525 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:44.852 22547== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:43.555 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:43.555+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:43.574 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:44.846 22555== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:43.575 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:44.848 22555== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:43.588 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:44.912 22547== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:43.613 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T15:59:43.613 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T15:59:43.622 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:44.894 22555== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:43.628 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:44.954 22547== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:43.635 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:43.635 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:43.645 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T15:59:43.650 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:44.977 22554== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:43.652 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:44.979 22554== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:43.657 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:43.654+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:43.690 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:43.691 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2024-08-30T15:59:43.691 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2024-08-30T15:59:43.691 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "peon", 2024-08-30T15:59:43.691 INFO:teuthology.orchestra.run.smithi050.stdout: "election_epoch": 6, 2024-08-30T15:59:43.691 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum": [ 2024-08-30T15:59:43.691 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2024-08-30T15:59:43.691 INFO:teuthology.orchestra.run.smithi050.stdout: 1, 2024-08-30T15:59:43.691 INFO:teuthology.orchestra.run.smithi050.stdout: 2 2024-08-30T15:59:43.691 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.691 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_age": 83, 2024-08-30T15:59:43.691 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2024-08-30T15:59:43.691 INFO:teuthology.orchestra.run.smithi050.stdout: "required_con": "2449958758054445060", 2024-08-30T15:59:43.692 INFO:teuthology.orchestra.run.smithi050.stdout: "required_mon": [ 2024-08-30T15:59:43.692 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2024-08-30T15:59:43.692 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2024-08-30T15:59:43.692 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2024-08-30T15:59:43.692 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2024-08-30T15:59:43.692 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2024-08-30T15:59:43.692 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2024-08-30T15:59:43.692 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2024-08-30T15:59:43.692 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2024-08-30T15:59:43.692 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2024-08-30T15:59:43.692 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2024-08-30T15:59:43.692 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.692 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_con": "4540138322906710015", 2024-08-30T15:59:43.693 INFO:teuthology.orchestra.run.smithi050.stdout: "quorum_mon": [ 2024-08-30T15:59:43.693 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2024-08-30T15:59:43.693 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2024-08-30T15:59:43.693 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2024-08-30T15:59:43.693 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2024-08-30T15:59:43.693 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2024-08-30T15:59:43.693 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2024-08-30T15:59:43.693 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2024-08-30T15:59:43.693 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2024-08-30T15:59:43.693 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2024-08-30T15:59:43.693 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2024-08-30T15:59:43.693 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "outside_quorum": [], 2024-08-30T15:59:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "extra_probe_peers": [], 2024-08-30T15:59:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_provider": [], 2024-08-30T15:59:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": { 2024-08-30T15:59:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 1, 2024-08-30T15:59:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "9d72e2e2-0554-42af-8e62-9b1a3a804112", 2024-08-30T15:59:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-08-30T15:57:29.788167Z", 2024-08-30T15:59:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-08-30T15:57:29.788167Z", 2024-08-30T15:59:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release": 18, 2024-08-30T15:59:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "min_mon_release_name": "reef", 2024-08-30T15:59:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "election_strategy": 1, 2024-08-30T15:59:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "disallowed_leaders: ": "", 2024-08-30T15:59:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false, 2024-08-30T15:59:43.694 INFO:teuthology.orchestra.run.smithi050.stdout: "tiebreaker_mon": "", 2024-08-30T15:59:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_ranks: ": "", 2024-08-30T15:59:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "features": { 2024-08-30T15:59:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "persistent": [ 2024-08-30T15:59:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "kraken", 2024-08-30T15:59:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "luminous", 2024-08-30T15:59:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "mimic", 2024-08-30T15:59:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "osdmap-prune", 2024-08-30T15:59:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "nautilus", 2024-08-30T15:59:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "octopus", 2024-08-30T15:59:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "pacific", 2024-08-30T15:59:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "elector-pinging", 2024-08-30T15:59:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "quincy", 2024-08-30T15:59:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "reef" 2024-08-30T15:59:43.696 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.696 INFO:teuthology.orchestra.run.smithi050.stdout: "optional": [] 2024-08-30T15:59:43.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.696 INFO:teuthology.orchestra.run.smithi050.stdout: "mons": [ 2024-08-30T15:59:43.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.696 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 0, 2024-08-30T15:59:43.696 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "a", 2024-08-30T15:59:43.696 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T15:59:43.696 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T15:59:43.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.696 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T15:59:43.696 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789", 2024-08-30T15:59:43.697 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2024-08-30T15:59:43.697 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.697 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.697 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6789/0", 2024-08-30T15:59:43.697 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6789/0", 2024-08-30T15:59:43.697 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2024-08-30T15:59:43.697 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T15:59:43.697 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2024-08-30T15:59:43.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.697 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 1, 2024-08-30T15:59:43.697 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "b", 2024-08-30T15:59:43.697 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T15:59:43.698 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T15:59:43.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.698 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T15:59:43.698 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6789", 2024-08-30T15:59:43.698 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2024-08-30T15:59:43.698 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.698 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.698 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6789/0", 2024-08-30T15:59:43.698 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6789/0", 2024-08-30T15:59:43.698 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2024-08-30T15:59:43.698 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T15:59:43.698 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2024-08-30T15:59:43.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.699 INFO:teuthology.orchestra.run.smithi050.stdout: "rank": 2, 2024-08-30T15:59:43.699 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "c", 2024-08-30T15:59:43.699 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T15:59:43.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T15:59:43.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.699 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T15:59:43.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790", 2024-08-30T15:59:43.699 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 0 2024-08-30T15:59:43.699 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.699 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6790/0", 2024-08-30T15:59:43.700 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6790/0", 2024-08-30T15:59:43.700 INFO:teuthology.orchestra.run.smithi050.stdout: "priority": 0, 2024-08-30T15:59:43.700 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T15:59:43.700 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "{}" 2024-08-30T15:59:43.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.700 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.700 INFO:teuthology.orchestra.run.smithi050.stdout: "feature_map": { 2024-08-30T15:59:43.700 INFO:teuthology.orchestra.run.smithi050.stdout: "mon": [ 2024-08-30T15:59:43.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.700 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2024-08-30T15:59:43.700 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "reef", 2024-08-30T15:59:43.700 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2024-08-30T15:59:43.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": [ 2024-08-30T15:59:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2024-08-30T15:59:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "reef", 2024-08-30T15:59:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 3 2024-08-30T15:59:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: "client": [ 2024-08-30T15:59:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2024-08-30T15:59:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "reef", 2024-08-30T15:59:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 4 2024-08-30T15:59:43.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.702 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T15:59:43.702 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": [ 2024-08-30T15:59:43.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:43.702 INFO:teuthology.orchestra.run.smithi050.stdout: "features": "0x3f01cfbffffdffff", 2024-08-30T15:59:43.702 INFO:teuthology.orchestra.run.smithi050.stdout: "release": "reef", 2024-08-30T15:59:43.702 INFO:teuthology.orchestra.run.smithi050.stdout: "num": 1 2024-08-30T15:59:43.702 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:43.702 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:43.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:43.702 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": false 2024-08-30T15:59:43.702 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:43.704 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:45.031 22554== Warning: unimplemented fcntl command: 1036 2024-08-30T15:59:43.705 INFO:tasks.mon_recovery:restarting each monitor in turn 2024-08-30T15:59:43.706 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-30T15:59:43.706 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T15:59:43.724 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T15:59:43.724 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T15:59:43.758 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:43.757+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:43.769 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T15:59:43.753+0000 a084640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 22531) UID: 0 2024-08-30T15:59:43.769 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T15:59:43.755+0000 a084640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-08-30T15:59:43.820 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:43.820 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:43.823 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T15:59:43.823 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T15:59:43.829 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T15:59:43.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-08-30T15:59:43.871 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:43.869+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:43.964 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T15:59:43.965 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T15:59:43.980 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:43.979+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:43.994 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:43.994 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:44.004 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T15:59:44.054 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T15:59:44.055 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T15:59:44.059 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:44.057+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:44.143 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T15:59:44.143 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T15:59:44.160 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:44.158+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:44.171 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:44.171 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:44.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T15:59:44.260 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T15:59:44.260 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T15:59:44.260 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:44.260+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:44.311 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:44.311 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T15:59:44.311 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T15:59:44.311 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:44.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T15:59:44.357 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T15:59:44.358 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T15:59:44.361 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:44.360+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:44.462 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:44.460+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:44.482 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:44.482 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T15:59:44.482 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T15:59:44.482 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T15:59:44.482 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:44.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T15:59:44.563 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:44.562+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:44.660 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:44.660 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T15:59:44.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T15:59:44.660 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T15:59:44.660 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:44.664 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:44.663+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:44.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T15:59:44.786 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:44.784+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:44.803 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:44.803 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T15:59:44.803 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T15:59:44.803 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:44.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T15:59:44.896 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:44.894+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:44.966 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:44.965+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:44.983 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:44.983 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T15:59:44.983 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T15:59:44.983 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T15:59:44.983 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:44.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T15:59:45.024 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T15:59:45.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-08-30T15:59:45.077 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:45.075+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:45.168 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:45.166+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:45.182 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T15:59:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T15:59:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T15:59:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T15:59:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T15:59:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 23.897317916999999, 2024-08-30T15:59:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T15:59:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T15:59:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T15:59:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T15:59:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T15:59:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T15:59:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T15:59:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T15:59:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T15:59:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T15:59:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T15:59:45.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T15:59:45.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T15:59:45.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T15:59:45.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T15:59:45.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T15:59:45.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T15:59:45.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T15:59:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T15:59:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T15:59:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T15:59:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T15:59:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T15:59:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T15:59:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T15:59:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T15:59:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T15:59:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T15:59:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T15:59:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T15:59:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T15:59:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 23.624390556000002, 2024-08-30T15:59:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T15:59:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T15:59:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T15:59:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T15:59:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T15:59:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T15:59:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T15:59:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T15:59:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T15:59:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T15:59:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T15:59:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T15:59:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T15:59:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T15:59:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T15:59:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T15:59:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T15:59:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T15:59:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T15:59:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T15:59:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T15:59:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T15:59:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T15:59:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T15:59:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T15:59:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T15:59:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T15:59:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T15:59:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T15:59:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:45.191 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:45.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T15:59:45.277 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:45.275+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:45.364 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T15:59:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T15:59:45.364 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:45.370 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:45.369+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:45.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T15:59:45.471 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:45.469+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:45.543 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:45.543 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T15:59:45.544 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T15:59:45.544 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T15:59:45.544 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:45.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T15:59:45.572 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:45.572+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:45.673 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:45.673+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:45.731 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:45.731 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T15:59:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T15:59:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T15:59:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T15:59:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T15:59:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.344493508999999, 2024-08-30T15:59:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T15:59:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T15:59:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T15:59:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T15:59:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T15:59:45.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T15:59:45.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:45.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T15:59:45.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T15:59:45.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:45.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T15:59:45.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T15:59:45.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T15:59:45.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T15:59:45.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T15:59:45.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T15:59:45.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T15:59:45.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T15:59:45.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T15:59:45.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T15:59:45.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T15:59:45.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T15:59:45.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T15:59:45.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T15:59:45.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T15:59:45.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T15:59:45.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T15:59:45.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T15:59:45.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T15:59:45.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T15:59:45.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T15:59:45.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:45.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:45.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T15:59:45.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T15:59:45.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T15:59:45.736 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:45.736 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:45.736 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:45.736 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:45.736 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:45.736 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:45.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T15:59:45.774 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:45.774+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:45.873 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:45.873 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T15:59:45.873 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T15:59:45.873 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:45.875 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:45.872+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:45.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T15:59:45.976 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:45.975+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:46.054 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:46.054 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T15:59:46.054 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T15:59:46.054 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T15:59:46.054 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:46.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T15:59:46.107 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:46.105+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:46.177 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:46.175+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:46.227 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:46.227 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T15:59:46.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T15:59:46.227 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T15:59:46.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.227 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T15:59:46.227 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T15:59:46.227 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.765123302999999, 2024-08-30T15:59:46.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T15:59:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T15:59:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T15:59:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T15:59:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T15:59:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T15:59:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T15:59:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T15:59:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T15:59:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T15:59:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T15:59:46.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T15:59:46.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T15:59:46.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T15:59:46.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T15:59:46.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T15:59:46.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T15:59:46.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T15:59:46.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T15:59:46.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T15:59:46.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T15:59:46.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T15:59:46.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T15:59:46.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T15:59:46.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T15:59:46.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T15:59:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T15:59:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T15:59:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T15:59:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T15:59:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T15:59:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T15:59:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T15:59:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T15:59:46.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T15:59:46.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T15:59:46.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T15:59:46.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T15:59:46.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T15:59:46.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T15:59:46.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T15:59:46.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T15:59:46.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T15:59:46.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T15:59:46.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T15:59:46.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T15:59:46.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T15:59:46.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T15:59:46.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T15:59:46.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T15:59:46.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T15:59:46.234 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.234 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:46.234 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.234 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T15:59:46.234 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T15:59:46.234 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.649256968, 2024-08-30T15:59:46.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T15:59:46.234 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T15:59:46.234 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T15:59:46.234 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T15:59:46.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T15:59:46.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T15:59:46.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T15:59:46.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T15:59:46.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T15:59:46.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T15:59:46.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T15:59:46.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T15:59:46.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T15:59:46.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T15:59:46.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T15:59:46.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T15:59:46.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T15:59:46.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T15:59:46.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T15:59:46.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T15:59:46.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T15:59:46.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T15:59:46.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T15:59:46.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T15:59:46.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T15:59:46.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T15:59:46.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T15:59:46.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T15:59:46.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T15:59:46.238 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.238 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:46.238 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.238 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T15:59:46.238 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T15:59:46.238 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.58966547, 2024-08-30T15:59:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T15:59:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T15:59:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T15:59:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T15:59:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T15:59:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T15:59:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T15:59:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T15:59:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T15:59:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T15:59:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T15:59:46.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T15:59:46.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T15:59:46.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T15:59:46.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T15:59:46.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T15:59:46.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T15:59:46.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T15:59:46.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T15:59:46.241 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T15:59:46.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T15:59:46.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T15:59:46.241 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T15:59:46.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T15:59:46.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T15:59:46.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T15:59:46.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T15:59:46.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T15:59:46.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T15:59:46.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T15:59:46.242 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.242 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:46.242 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.242 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T15:59:46.243 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T15:59:46.243 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.562966783, 2024-08-30T15:59:46.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T15:59:46.243 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T15:59:46.243 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T15:59:46.243 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T15:59:46.243 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T15:59:46.243 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T15:59:46.243 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T15:59:46.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.243 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T15:59:46.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T15:59:46.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T15:59:46.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T15:59:46.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T15:59:46.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T15:59:46.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T15:59:46.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T15:59:46.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T15:59:46.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T15:59:46.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T15:59:46.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T15:59:46.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T15:59:46.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T15:59:46.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T15:59:46.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T15:59:46.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T15:59:46.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T15:59:46.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T15:59:46.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T15:59:46.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T15:59:46.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T15:59:46.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T15:59:46.246 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.246 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:46.246 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.247 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T15:59:46.247 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T15:59:46.247 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.695350255000001, 2024-08-30T15:59:46.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T15:59:46.247 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T15:59:46.247 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T15:59:46.247 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T15:59:46.247 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T15:59:46.247 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T15:59:46.247 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T15:59:46.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.247 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T15:59:46.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.248 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T15:59:46.248 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T15:59:46.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.248 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T15:59:46.248 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T15:59:46.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.248 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T15:59:46.248 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T15:59:46.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T15:59:46.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T15:59:46.249 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T15:59:46.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T15:59:46.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T15:59:46.249 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T15:59:46.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T15:59:46.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T15:59:46.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T15:59:46.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T15:59:46.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T15:59:46.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T15:59:46.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T15:59:46.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T15:59:46.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T15:59:46.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T15:59:46.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T15:59:46.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T15:59:46.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T15:59:46.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T15:59:46.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T15:59:46.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T15:59:46.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T15:59:46.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T15:59:46.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T15:59:46.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T15:59:46.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T15:59:46.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T15:59:46.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T15:59:46.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T15:59:46.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T15:59:46.254 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.254 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:46.254 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T15:59:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T15:59:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.536103176000001, 2024-08-30T15:59:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T15:59:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T15:59:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T15:59:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T15:59:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T15:59:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T15:59:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T15:59:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T15:59:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T15:59:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T15:59:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T15:59:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T15:59:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T15:59:46.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T15:59:46.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T15:59:46.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T15:59:46.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T15:59:46.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T15:59:46.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T15:59:46.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T15:59:46.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T15:59:46.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T15:59:46.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T15:59:46.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T15:59:46.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T15:59:46.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T15:59:46.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T15:59:46.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T15:59:46.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T15:59:46.258 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.259 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:46.259 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.259 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T15:59:46.259 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T15:59:46.259 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.259929110000002, 2024-08-30T15:59:46.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T15:59:46.259 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T15:59:46.259 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T15:59:46.259 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T15:59:46.259 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T15:59:46.259 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T15:59:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T15:59:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T15:59:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T15:59:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T15:59:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T15:59:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T15:59:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T15:59:46.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T15:59:46.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T15:59:46.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T15:59:46.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T15:59:46.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T15:59:46.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T15:59:46.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T15:59:46.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T15:59:46.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.262 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T15:59:46.262 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T15:59:46.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T15:59:46.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.262 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T15:59:46.262 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T15:59:46.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T15:59:46.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.262 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T15:59:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T15:59:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T15:59:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T15:59:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T15:59:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 18.601637531000002, 2024-08-30T15:59:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T15:59:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T15:59:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T15:59:46.263 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T15:59:46.264 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T15:59:46.264 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T15:59:46.264 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T15:59:46.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.264 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T15:59:46.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.264 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T15:59:46.264 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T15:59:46.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.264 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T15:59:46.264 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T15:59:46.265 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.265 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T15:59:46.265 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T15:59:46.265 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T15:59:46.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.265 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T15:59:46.265 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T15:59:46.265 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T15:59:46.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T15:59:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T15:59:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T15:59:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T15:59:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T15:59:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T15:59:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T15:59:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T15:59:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T15:59:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T15:59:46.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T15:59:46.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T15:59:46.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T15:59:46.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T15:59:46.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T15:59:46.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T15:59:46.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T15:59:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T15:59:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T15:59:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T15:59:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T15:59:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T15:59:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T15:59:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T15:59:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T15:59:46.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T15:59:46.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T15:59:46.269 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.269 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:46.269 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.269 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T15:59:46.269 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T15:59:46.269 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 18.505724247, 2024-08-30T15:59:46.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T15:59:46.269 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T15:59:46.269 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T15:59:46.270 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T15:59:46.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T15:59:46.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T15:59:46.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T15:59:46.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T15:59:46.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T15:59:46.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T15:59:46.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T15:59:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T15:59:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T15:59:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T15:59:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T15:59:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T15:59:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T15:59:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T15:59:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.272 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T15:59:46.272 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T15:59:46.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T15:59:46.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.272 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T15:59:46.272 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T15:59:46.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T15:59:46.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.272 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T15:59:46.272 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T15:59:46.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T15:59:46.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T15:59:46.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T15:59:46.273 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T15:59:46.273 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T15:59:46.273 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T15:59:46.273 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.273 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:46.273 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.273 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T15:59:46.273 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T15:59:46.273 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:46.274 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T15:59:46.278 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:46.277+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:46.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T15:59:46.418 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:46.416+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:46.419 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T15:59:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T15:59:46.419 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T15:59:46.419 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T15:59:46.429 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T15:59:46.479 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:46.477+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:46.559 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T15:59:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T15:59:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T15:59:46.559 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T15:59:46.559 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T15:59:46.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T15:59:46.580 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:46.578+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:46.610 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T15:59:46.611 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T15:59:46.682 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:46.679+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:46.697 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T15:59:46.698 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T15:59:46.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T15:59:46.698 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T15:59:46.698 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T15:59:46.708 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T15:59:46.709 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T15:59:46.709 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T15:59:46.783 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:46.780+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:46.796 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T15:59:46.796 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T15:59:46.835 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T15:59:46.835 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T15:59:46.835 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T15:59:46.835 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T15:59:46.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T15:59:46.883 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:46.881+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:46.888 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T15:59:46.888 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T15:59:46.984 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T15:59:46.984 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T15:59:46.984 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T15:59:46.984 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T15:59:46.984 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T15:59:46.988 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:46.986+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:46.988 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T15:59:46.988 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T15:59:46.993 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T15:59:47.073 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T15:59:47.073 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T15:59:47.115 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:47.113+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:47.186 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:47.184+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:47.191 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:47.191 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:47.191 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T15:59:47.192 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T15:59:47.200 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T15:59:47.276 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T15:59:47.276 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T15:59:47.287 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:47.285+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:47.371 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:47.372 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:47.382 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T15:59:47.424 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:47.422+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:47.489 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:47.487+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:47.550 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:47.550 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:47.559 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T15:59:47.602 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:47.599+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:47.691 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:47.690+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:47.723 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:47.723 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:47.733 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T15:59:47.791 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:47.790+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:47.842 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-08-30T15:59:47.859 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:47.859 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:47.867 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T15:59:47.912 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:47.910+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:47.990 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:47.990 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:47.993 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:47.992+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:47.999 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T15:59:48.094 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:48.093+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:48.162 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:48.162 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:48.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T15:59:48.217 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:48.216+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:48.297 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:48.296+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:48.312 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:48.312 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:48.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T15:59:48.399 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:48.397+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:48.491 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:48.492 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:48.499 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:48.498+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:48.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T15:59:48.600 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:48.599+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:48.629 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:48.629 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:48.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T15:59:48.701 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:48.700+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:48.768 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:48.769 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:48.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T15:59:48.802 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:48.800+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:48.903 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:48.902+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:48.948 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:48.948 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:48.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T15:59:49.004 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:49.002+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:49.088 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:49.088 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:49.098 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T15:59:49.142 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:49.140+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:49.215 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:49.214+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:49.225 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:49.225 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:49.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T15:59:49.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T15:59:49.320 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:49.318+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:49.406 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:49.407 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:49.409 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:49.407+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:49.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T15:59:49.510 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:49.508+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:49.542 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:49.542 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:49.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T15:59:49.611 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:49.609+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:49.679 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:49.680 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:49.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T15:59:49.733 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:49.732+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:49.807 INFO:tasks.ceph.mon.a:Stopped 2024-08-30T15:59:49.807 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2024-08-30T15:59:49.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T15:59:49.820 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:49.821 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:49.821 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:49.820+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:49.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T15:59:49.914 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:49.913+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:50.001 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:50.001 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:50.010 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T15:59:50.014 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:50.012+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:50.115 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:50.113+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:50.148 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:50.148 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:50.157 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T15:59:50.216 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:50.214+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:50.284 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:50.285 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:50.293 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T15:59:50.316 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:50.315+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:50.417 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:50.416+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:50.461 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:50.461 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:50.470 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T15:59:50.555 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:50.553+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:50.619 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:50.618+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:50.633 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:50.633 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:50.642 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T15:59:50.721 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:50.720+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:50.804 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:50.804 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:50.813 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T15:59:50.857 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:50.855+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:50.923 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:50.922+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:50.937 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:50.937 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:50.947 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T15:59:51.024 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:51.023+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:51.112 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:51.113 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:51.121 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T15:59:51.124 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:51.123+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:51.225 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:51.224+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:51.249 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:51.249 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:51.258 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T15:59:51.326 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:51.325+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:51.422 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:51.423 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:51.427 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:51.426+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:51.431 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T15:59:51.528 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:51.527+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:51.556 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:51.556 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:51.566 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T15:59:51.651 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:51.649+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:51.731 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:51.729+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:51.732 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:51.732 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:51.741 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T15:59:51.832 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:51.830+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:51.906 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:51.907 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:51.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T15:59:51.961 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:51.960+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:52.032 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:52.031+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:52.057 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:52.058 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:52.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T15:59:52.133 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:52.132+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:52.235 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:52.233+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:52.236 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:52.236 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:52.245 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T15:59:52.336 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:52.334+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:52.372 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:52.372 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:52.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T15:59:52.437 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:52.435+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:52.510 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:52.511 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:52.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T15:59:52.561 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:52.560+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:52.638 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:52.637+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:52.692 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:52.692 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:52.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T15:59:52.740 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:52.738+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:52.830 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:52.831 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:52.840 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T15:59:52.841 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:52.839+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:52.941 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:52.940+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:53.008 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:53.008 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:53.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T15:59:53.062 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:53.061+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:53.143 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:53.142+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:53.150 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:53.150 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:53.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T15:59:53.244 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:53.242+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:53.327 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:53.327 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:53.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T15:59:53.345 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:53.343+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:53.446 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:53.444+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:53.466 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:53.467 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:53.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T15:59:53.547 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:53.545+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:53.605 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:53.606 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:53.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T15:59:53.648 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:53.646+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:53.746 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:53.747 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:53.748 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:53.747+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:53.755 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T15:59:53.849 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:53.848+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:53.894 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:53.894 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:53.904 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T15:59:53.950 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:53.948+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:54.030 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:54.030 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:54.040 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T15:59:54.050 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-08-30T15:59:54.051 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:54.050+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:54.151 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:54.150+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:54.205 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:54.205 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:54.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T15:59:54.255 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:54.253+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:54.353 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:54.352+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:54.382 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:54.382 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:54.392 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T15:59:54.455 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:54.454+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:54.555 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:54.554+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:54.558 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:54.558 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:54.567 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T15:59:54.656 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:54.655+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:54.690 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:54.690 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:54.699 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T15:59:54.757 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:54.755+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:54.824 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:54.824 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:54.833 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T15:59:54.876 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:54.873+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:54.958 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:54.956+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:54.997 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:54.997 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:55.006 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T15:59:55.058 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:55.057+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:55.159 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:55.158+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:55.172 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:55.172 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:55.182 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T15:59:55.261 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:55.259+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:55.351 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:55.351 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:55.360 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T15:59:55.362 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:55.360+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:55.463 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:55.462+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:55.486 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:55.486 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:55.496 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T15:59:55.564 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:55.563+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:55.621 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:55.621 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:55.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T15:59:55.665 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:55.663+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:55.766 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:55.764+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:55.766 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:55.766 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:55.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T15:59:55.867 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:55.865+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:55.902 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:55.902 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:55.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T15:59:55.968 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:55.967+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:56.039 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:56.039 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:56.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T15:59:56.090 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:56.088+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:56.169 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:56.168+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:56.217 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:56.217 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:56.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T15:59:56.270 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:56.268+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:56.355 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:56.356 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:56.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T15:59:56.370 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:56.369+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:56.472 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:56.470+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:56.494 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:56.495 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:56.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T15:59:56.572 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:56.571+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:56.636 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:56.636 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:56.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T15:59:56.689 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:56.687+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:56.773 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:56.772+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:56.819 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:56.820 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:56.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T15:59:56.874 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:56.873+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:56.976 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:56.974+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:56.996 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:56.996 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:57.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T15:59:57.077 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:57.075+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:57.137 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:57.137 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:57.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T15:59:57.189 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:57.186+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:57.277 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:57.276+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:57.315 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:57.315 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:57.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T15:59:57.378 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:57.377+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:57.479 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:57.477+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:57.493 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:57.493 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:57.504 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T15:59:57.580 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:57.579+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:57.641 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:57.642 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:57.651 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T15:59:57.696 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:57.694+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:57.776 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:57.776 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:57.781 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:57.780+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:57.784 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T15:59:57.882 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:57.881+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:57.952 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:57.952 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:57.962 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T15:59:57.983 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:57.982+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:58.084 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:58.083+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:58.127 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:58.127 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:58.136 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T15:59:58.185 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:58.184+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:58.259 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:58.259 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:58.268 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T15:59:58.286 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:58.285+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:58.387 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:58.386+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:58.396 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:58.396 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:58.404 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T15:59:58.489 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T15:59:58.487+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T15:59:58.571 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:58.571 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:58.580 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T15:59:58.590 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:58.589+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:58.694 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:58.692+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:58.706 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:58.707 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:58.716 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T15:59:58.801 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:58.799+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:58.886 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:58.886 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:58.893 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T15:59:58.892+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T15:59:58.896 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T15:59:58.965 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T15:59:58.965 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T15:59:58.994 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T15:59:58.993+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T15:59:59.058 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T15:59:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T15:59:59.063 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:59.063 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:59.072 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T15:59:59.095 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:59.094+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:59.160 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T15:59:59.161 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T15:59:59.196 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:59.194+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:59.199 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:59.199 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:59.208 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T15:59:59.252 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T15:59:59.252 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T15:59:59.296 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:59.295+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:59.341 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T15:59:59.430 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T15:59:59.431 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":8,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T15:59:59.432 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2024-08-30T15:59:59.432 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-30T15:59:59.432 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-08-30T15:59:59.433 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T15:59:59.433 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T15:59:59.434 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T15:59:59.434 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T15:59:59.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T15:59:59.435 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:59.433+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:59.475 INFO:tasks.ceph.mon.a:Started 2024-08-30T15:59:59.475 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-08-30T15:59:59.475 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T15:59:59.515 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T15:59:59.516 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T15:59:59.535 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T15:59:59.534+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T15:59:59.627 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T15:59:59.628 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T15:59:59.636 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T15:59:59.635+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T15:59:59.642 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T15:59:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T15:59:59.643 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:59.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T15:59:59.737 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T15:59:59.735+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T15:59:59.822 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:59.822 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T15:59:59.822 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T15:59:59.822 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T15:59:59.822 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:59.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T15:59:59.874 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T15:59:59.873+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T15:59:59.939 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T15:59:59.938+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T15:59:59.981 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T15:59:59.981 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T15:59:59.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T15:59:59.981 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T15:59:59.981 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T15:59:59.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-08-30T16:00:00.039 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.0 to 1 (10000) 2024-08-30T16:00:00.039 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:00.038+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:00.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-08-30T16:00:00.140 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:00.139+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:00.150 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:00.150 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:00.150 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:00.151 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:00.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:00.188 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":8,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:00:00.246 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:00.245+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:00.322 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:00.322 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:00.322 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:00.322 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:00.322 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:00.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:00.341 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:00.340+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:00.443 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:00.441+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:00.508 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:00.508 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:00.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:00.508 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:00.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.509 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:00.509 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:00.509 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 39.233658931999997, 2024-08-30T16:00:00.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:00:00.509 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:00.509 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:00.509 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:00.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:00.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:00.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:00.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:00.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:00.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:00.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:00.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:00:00.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:00:00.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:00.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:00:00.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:00:00.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:00.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:00:00.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:00:00.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:00.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:00:00.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:00:00.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:00.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:00:00.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:00:00.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:00.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:00:00.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:00:00.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:00:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:00:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 38.960731570999997, 2024-08-30T16:00:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:00:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:00.514 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:00.514 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:00.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:00.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:00.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:00.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:00.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:00.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:00.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:00.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:00:00.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:00:00.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:00.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:00:00.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:00:00.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:00.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:00:00.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:00:00.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:00.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:00:00.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:00:00.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:00.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:00:00.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:00:00.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:00.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:00:00.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:00:00.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:00.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:00:00.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:00:00.517 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:00.517 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:00.517 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:00.517 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:00.517 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:00.517 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:00.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:00.543 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:00.542+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:00.644 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:00.643+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:00.645 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:00.646 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:00.646 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:00.646 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:00.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:00.746 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:00.744+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:00.783 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:00.783 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:00.784 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:00.784 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:00.784 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:00.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:00.846 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:00.845+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:00.923 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:00.924 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:00.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:00.924 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:00.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.924 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:00:00.924 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:00.924 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.546095358000002, 2024-08-30T16:00:00.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:00:00.924 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:00.924 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:00.924 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:00.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:00.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:00.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:00.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:00.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:00.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:00.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:00.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:00:00.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:00:00.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:00.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:00:00.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:00:00.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:00.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:00:00.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:00:00.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:00.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:00:00.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:00:00.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:00.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:00:00.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:00:00.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:00.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:00:00.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:00:00.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:00.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:00:00.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:00:00.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:00.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:00.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:00.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:00:00.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:00:00.929 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:00.929 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:00.929 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:00.929 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:00.929 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:00.929 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:00.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:00.977 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:00.976+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:00.980 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.1 to 1 (10000) 2024-08-30T16:00:00.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-08-30T16:00:01.084 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:01.082+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:01.088 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:01.088 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:01.088 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:01.088 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:01.098 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:01.148 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:01.146+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:01.249 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:01.247+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:01.288 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:01.288 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:01.288 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:01.288 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:01.288 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:01.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:01.350 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:01.348+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:01.451 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:01.449+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:01.455 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:01.455 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:01.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:01.455 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:01.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.455 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:01.455 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:01.455 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 37.005616687, 2024-08-30T16:00:01.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:00:01.455 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:01.456 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:01.456 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:01.456 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:01.456 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:01.456 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:01.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.456 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:01.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:01.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:01.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:00:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:00:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:00:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:00:01.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:01.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:00:01.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:00:01.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:01.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:00:01.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:00:01.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:01.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:00:01.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:00:01.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:00:01.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:00:01.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:00:01.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:01.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:00:01.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:00:01.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:01.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:00:01.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:00:01.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:00:01.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:00:01.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:00:01.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:01.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:00:01.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:00:01.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:01.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:00:01.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:00:01.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:01.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:00:01.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:00:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:00:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:00:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:00:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:01.463 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.889750352, 2024-08-30T16:00:01.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:00:01.464 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:01.464 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:01.464 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:01.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:01.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:01.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:01.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:01.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:01.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:00:01.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:01.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:01.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:00:01.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:00:01.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:00:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:00:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:00:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:00:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:00:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:00:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:01.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:00:01.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:00:01.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:01.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:00:01.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:00:01.467 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.467 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:01.467 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.468 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:01.468 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:01.468 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.830158853999997, 2024-08-30T16:00:01.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:00:01.468 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:01.468 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:01.468 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:01.468 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:01.468 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:01.468 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:00:01.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.468 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:01.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:01.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:01.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:01.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:01.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:01.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:00:01.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:00:01.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:01.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:00:01.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:00:01.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:01.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:00:01.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:00:01.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:01.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:00:01.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:00:01.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:01.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:00:01.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:00:01.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:01.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:00:01.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:00:01.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.472 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:01.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.472 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:00:01.472 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:01.472 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.803460166999997, 2024-08-30T16:00:01.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:00:01.473 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:01.473 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:01.473 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:01.473 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:01.473 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:01.473 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:01.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.473 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:01.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:01.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:01.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:01.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:01.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:01.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:00:01.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:01.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:01.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:00:01.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:00:01.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:01.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:00:01.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:00:01.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:01.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:00:01.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:00:01.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:01.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:00:01.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:00:01.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:01.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:00:01.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:00:01.476 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.476 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:01.476 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.477 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:01.477 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:01.477 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 34.935843638999998, 2024-08-30T16:00:01.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:00:01.477 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:01.477 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:01.477 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:01.477 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:01.477 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:01.477 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:00:01.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.477 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:01.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:01.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:01.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:01.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:01.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:01.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:00:01.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:01.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:00:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:00:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:00:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:00:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:00:01.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:00:01.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:01.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:00:01.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:00:01.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:01.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:00:01.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:00:01.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:01.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:00:01.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:00:01.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:01.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:00:01.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:00:01.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:01.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:00:01.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:00:01.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:01.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:00:01.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:00:01.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:01.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:00:01.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:00:01.483 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.483 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:01.483 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.483 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:00:01.483 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:01.483 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 34.776596560000002, 2024-08-30T16:00:01.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:00:01.483 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:01.483 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:01.483 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:01.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:01.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:01.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:01.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:01.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:01.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:00:01.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:00:01.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:00:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:00:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:00:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:00:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:00:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:00:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:01.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:00:01.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:00:01.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:01.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:00:01.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:00:01.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:00:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:00:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 34.500422493999999, 2024-08-30T16:00:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:00:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:01.487 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:01.488 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:01.488 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:01.488 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:01.488 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:01.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.488 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:01.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:01.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:01.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:01.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:01.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:01.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:00:01.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:01.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:01.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:00:01.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:00:01.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:01.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:00:01.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:00:01.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:01.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:00:01.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:00:01.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:01.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:00:01.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:00:01.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:01.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:00:01.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:00:01.491 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.491 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:01.491 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.491 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:01.492 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:01.492 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.842130914999998, 2024-08-30T16:00:01.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:00:01.492 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:01.492 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:01.492 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:01.492 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:01.492 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:01.492 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:00:01.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.492 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:01.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:01.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:01.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:01.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:01.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:01.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:00:01.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:01.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:01.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:00:01.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:00:01.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:01.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:00:01.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:00:01.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:01.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:00:01.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:00:01.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:01.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:00:01.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:00:01.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:01.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:00:01.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:00:01.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:01.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:00:01.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:00:01.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:01.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:00:01.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:00:01.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:01.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:00:01.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:00:01.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:01.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:00:01.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:00:01.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:01.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:00:01.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:00:01.498 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.498 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:01.498 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.499 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:00:01.499 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:01.499 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.746217631, 2024-08-30T16:00:01.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:00:01.499 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:01.499 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:01.499 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:01.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:01.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:01.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:01.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:01.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:01.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:01.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:00:01.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:01.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:01.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:00:01.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:00:01.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:01.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:00:01.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:00:01.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:01.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:00:01.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:00:01.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:01.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:00:01.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:00:01.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:01.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:00:01.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:00:01.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:01.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:01.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:01.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:00:01.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:00:01.502 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.502 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:01.502 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.503 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:01.503 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:01.503 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:01.504 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:01.552 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:01.550+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:01.632 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:01.633 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:01.633 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:01.633 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:01.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:00:01.642 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:01.684 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:01.682+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:01.754 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:01.752+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:01.821 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:01.821 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:01.830 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:00:01.854 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:01.853+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:01.890 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:00:01.890 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:00:01.956 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:01.954+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:01.983 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:00:01.983 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:00:01.998 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:01.998 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:01.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:01.998 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:00:01.998 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:02.008 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:00:02.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-08-30T16:00:02.074 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:00:02.074 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:00:02.075 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:02.073+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:02.105 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.2 to 1 (10000) 2024-08-30T16:00:02.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-08-30T16:00:02.157 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:02.156+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:02.181 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:02.181 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:02.181 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:02.181 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:02.189 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:00:02.190 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:00:02.192 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:00:02.259 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:02.257+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:02.276 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:00:02.276 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:00:02.359 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:02.358+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:02.367 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:00:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:00:02.406 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:02.406 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:02.416 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:00:02.457 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:00:02.458 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:00:02.465 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:02.463+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:02.544 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:02.544 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:02.553 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:00:02.560 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:02.559+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:02.563 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:00:02.563 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:00:02.661 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:02.659+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:02.718 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:02.718 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:02.728 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:00:02.762 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:02.761+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:02.863 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:02.861+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:02.892 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:02.892 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:02.901 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:00:02.964 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:02.963+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:03.024 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:03.024 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:03.034 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:00:03.065 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:03.063+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:03.163 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:03.163 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:03.166 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:03.165+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:03.172 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:00:03.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:00:03.257 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.3 to 1 (10000) 2024-08-30T16:00:03.267 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:03.266+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:03.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-08-30T16:00:03.326 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-08-30T16:00:03.341 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:03.341 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:03.350 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:00:03.392 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:03.390+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:03.469 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:03.467+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:03.540 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:03.540 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:03.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:00:03.569 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:03.567+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:03.670 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:03.669+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:03.719 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:03.719 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:03.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:00:03.776 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:03.774+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:03.873 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:03.871+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:03.925 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:03.925 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:03.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:00:03.937 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":8,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:00:03.974 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:03.972+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:04.064 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:04.064 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:04.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:04.074 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:04.073+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:04.175 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:04.173+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:04.203 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:04.204 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:04.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:04.276 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:04.275+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:04.377 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:04.375+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:04.379 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:04.379 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:04.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:04.422 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.4 to 1 (10000) 2024-08-30T16:00:04.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-08-30T16:00:04.478 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:04.476+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:04.550 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:04.551 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:04.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:04.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:00:04.605 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:04.604+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:04.699 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:04.697+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:04.713 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:04.713 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:04.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:04.780 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:04.778+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:04.873 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:00:04.874 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:00:04.899 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:04.897+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:04.920 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:04.930 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:04.965 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:00:04.965 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:00:05.009 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:05.008+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:05.084 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:05.083+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:05.093 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:00:05.093 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:00:05.133 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:05.133 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:05.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:05.133 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:05.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.133 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:00:05.133 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:05.133 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 39.755515608000003, 2024-08-30T16:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:05.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:00:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:00:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:00:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:00:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:00:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:00:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:00:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:00:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:00:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:00:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:00:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:00:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:05.138 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:05.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:05.185 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:05.184+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:05.219 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:00:05.220 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:00:05.286 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:05.285+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:05.314 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:05.314 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:05.314 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:05.314 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:05.315 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:00:05.315 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:00:05.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:05.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-08-30T16:00:05.398 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:00:05.399 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:00:05.399 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:05.398+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:05.407 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.5 to 1 (10000) 2024-08-30T16:00:05.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-08-30T16:00:05.488 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:05.487+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:05.511 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:00:05.511 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:00:05.545 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:05.546 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:05.546 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:05.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:05.599 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:05.598+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:05.609 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:00:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:00:05.691 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:05.690+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:05.751 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:05.751 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:05.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:05.751 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:05.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.751 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:05.751 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:05.752 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 41.302330564000002, 2024-08-30T16:00:05.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:00:05.752 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:05.752 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:05.752 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:05.752 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:05.752 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:05.752 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:05.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.752 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:05.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:05.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:05.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:05.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:05.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:05.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:00:05.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:00:05.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:05.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:00:05.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:00:05.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:05.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:00:05.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:00:05.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:05.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:00:05.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:00:05.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:05.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:00:05.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:00:05.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:00:05.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:00:05.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:00:05.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:05.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:00:05.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:00:05.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:05.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:00:05.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:00:05.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:00:05.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:00:05.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:00:05.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:05.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:00:05.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:00:05.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:05.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:00:05.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:00:05.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:05.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:00:05.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:00:05.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:05.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:00:05.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:00:05.758 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.758 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:05.758 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.758 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:00:05.758 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:05.758 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 40.186464229000002, 2024-08-30T16:00:05.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:00:05.759 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:05.759 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:05.759 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:05.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:05.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:05.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:05.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:05.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:05.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:00:05.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:05.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:05.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:00:05.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:00:05.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:05.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:00:05.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:00:05.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:05.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:00:05.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:00:05.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:05.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:00:05.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:00:05.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:00:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:00:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:00:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:00:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 40.126872730999999, 2024-08-30T16:00:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:00:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:00:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:00:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:00:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:00:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:00:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:00:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:00:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:00:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:00:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:00:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:00:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:00:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:00:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.767 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:00:05.767 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:05.767 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 40.100174043999999, 2024-08-30T16:00:05.767 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:00:05.767 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:05.767 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:05.767 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:05.767 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:05.767 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:05.767 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:05.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.767 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:05.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:05.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:05.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:05.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:05.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:05.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:00:05.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:05.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.769 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:05.769 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:00:05.769 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:00:05.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.769 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:05.769 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:00:05.769 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:00:05.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.769 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:05.770 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:00:05.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:00:05.770 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.770 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.770 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:05.770 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:00:05.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:00:05.770 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.770 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.770 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:05.770 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:00:05.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:00:05.770 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.771 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:05.771 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.771 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.771 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.771 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:05.771 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:05.771 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 39.232557516, 2024-08-30T16:00:05.771 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:00:05.771 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:05.771 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:05.771 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:05.771 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:05.771 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:05.771 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:00:05.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.772 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:05.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.772 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:05.772 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:05.772 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.772 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:05.772 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:05.772 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.773 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:05.773 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:00:05.773 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:05.773 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.773 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.773 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:05.773 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:00:05.773 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:00:05.773 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.773 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.773 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:05.773 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:00:05.773 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:00:05.774 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.774 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:05.774 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:00:05.774 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:00:05.774 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.774 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:05.774 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:00:05.774 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:00:05.774 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.774 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:05.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:00:05.775 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:00:05.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:05.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:00:05.775 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:00:05.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:05.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:00:05.775 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:00:05.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.776 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:05.776 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:00:05.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:00:05.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.776 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:05.776 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:00:05.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:00:05.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.776 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:05.776 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:00:05.777 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:00:05.777 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.777 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:05.777 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.777 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:00:05.777 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:05.777 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 39.073310437000004, 2024-08-30T16:00:05.777 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:00:05.777 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:05.777 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:05.777 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:05.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.778 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:05.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:05.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.778 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:05.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:05.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.778 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:05.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:00:05.779 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:00:05.779 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:05.779 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:00:05.779 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:00:05.779 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:05.779 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:00:05.779 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:00:05.779 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:05.780 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:00:05.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:00:05.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.780 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.780 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:05.780 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:00:05.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:00:05.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.780 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.780 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:05.780 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:00:05.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:00:05.781 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.781 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.781 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:05.781 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:00:05.781 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:00:05.781 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.781 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:05.781 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.781 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.781 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.781 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:05.781 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 38.797136371000001, 2024-08-30T16:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:05.782 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:00:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:00:05.784 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:00:05.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:05.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:00:05.784 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:00:05.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:05.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:00:05.784 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:00:05.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.785 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:05.785 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:00:05.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:00:05.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.785 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:05.785 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:00:05.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:00:05.786 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.786 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:05.786 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.786 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:05.786 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:05.786 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 38.138844792, 2024-08-30T16:00:05.786 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:00:05.786 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:05.786 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:05.786 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:05.786 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:05.787 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:05.787 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:00:05.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.787 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:05.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:05.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:05.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:05.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:05.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.788 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:05.788 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:00:05.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:05.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.788 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:05.788 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:00:05.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:00:05.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.788 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:05.789 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:00:05.789 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:00:05.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.789 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:05.789 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:00:05.789 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:00:05.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.789 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:05.789 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:00:05.789 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:00:05.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.790 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:05.790 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:00:05.790 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:00:05.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.790 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:05.790 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:00:05.790 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:00:05.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.790 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:05.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:00:05.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:00:05.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.791 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:05.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:00:05.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:00:05.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.791 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:05.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:00:05.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:00:05.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:00:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:00:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:00:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 38.042931508000002, 2024-08-30T16:00:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:00:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:05.793 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:05.793 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:05.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:05.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:05.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:05.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:05.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:05.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:05.794 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:00:05.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:05.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:05.794 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:00:05.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:00:05.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:05.794 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:00:05.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:00:05.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:05.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:00:05.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:00:05.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:05.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:00:05.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:00:05.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:05.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:00:05.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:00:05.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:05.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:05.796 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:05.796 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:00:05.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:00:05.796 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.796 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:05.796 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.796 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:05.796 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:05.796 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:05.798 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:05.798 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:05.797+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:05.899 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:05.898+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:05.927 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:05.928 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:05.928 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:05.937 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:06.000 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:05.998+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:06.062 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:06.062 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:06.062 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:06.063 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:06.073 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:00:06.100 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:06.099+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:06.201 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:06.199+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:06.244 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:06.244 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:06.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:06.244 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:00:06.244 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:06.253 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:00:06.337 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:06.335+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:06.402 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:06.400+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:06.417 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:06.417 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:06.417 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:06.426 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:00:06.510 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:06.508+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:06.593 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-08-30T16:00:06.602 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.6 to 1 (10000) 2024-08-30T16:00:06.608 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:06.606+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:06.609 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:06.609 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:06.610 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:06.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-08-30T16:00:06.618 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:00:06.709 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:06.708+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:06.810 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:06.809+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:06.850 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:06.850 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:06.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:06.851 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:06.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.851 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:06.851 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:06.851 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.975893790000001, 2024-08-30T16:00:06.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:00:06.851 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.851 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.851 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.851 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.851 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.852 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:00:06.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.852 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:06.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:06.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:00:06.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:00:06.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:00:06.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:00:06.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:00:06.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:00:06.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:00:06.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:00:06.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:00:06.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:00:06.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:00:06.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:00:06.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:00:06.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:00:06.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:00:06.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:00:06.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:00:06.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:00:06.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:00:06.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:00:06.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:00:06.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:00:06.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:00:06.857 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.857 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.857 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.857 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:06.858 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:06.858 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.059477770000001, 2024-08-30T16:00:06.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:00:06.858 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.858 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.858 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.858 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.858 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.858 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:00:06.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.858 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:06.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:06.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:00:06.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:06.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:00:06.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:00:06.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:00:06.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:00:06.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:00:06.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:00:06.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:00:06.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:00:06.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:00:06.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:00:06.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:00:06.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:00:06.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:00:06.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:00:06.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:00:06.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:00:06.863 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.863 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.863 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.863 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:06.863 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:06.863 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.057773216999998, 2024-08-30T16:00:06.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:00:06.863 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.863 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.863 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.864 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.864 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.864 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:00:06.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.864 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:06.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:06.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:00:06.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:00:06.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:00:06.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:00:06.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:00:06.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:00:06.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:00:06.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:00:06.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:00:06.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:00:06.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:00:06.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:00:06.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:00:06.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:00:06.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:00:06.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:00:06.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:00:06.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:00:06.868 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.868 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.868 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.869 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.869 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:06.869 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 54.638319410000001, 2024-08-30T16:00:06.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:00:06.869 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.869 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.869 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.869 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.869 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.869 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:00:06.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.869 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:06.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:06.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:00:06.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:00:06.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:00:06.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:00:06.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:00:06.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:00:06.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:00:06.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:00:06.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:00:06.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:00:06.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:00:06.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:00:06.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:00:06.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:00:06.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:00:06.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:00:06.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:00:06.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:00:06.874 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.874 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.874 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.874 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.874 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:06.874 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 54.621431235999999, 2024-08-30T16:00:06.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:00:06.874 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.874 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.875 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.875 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.875 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.875 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:00:06.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.875 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:06.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:06.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:00:06.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:00:06.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:00:06.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:00:06.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:00:06.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:00:06.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:00:06.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:00:06.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:00:06.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:00:06.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:00:06.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:00:06.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:00:06.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:00:06.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:00:06.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:00:06.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:00:06.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:00:06.879 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.879 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.879 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.879 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.880 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:06.880 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 54.608431166999999, 2024-08-30T16:00:06.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:00:06.880 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.880 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.880 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.880 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.880 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.880 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:00:06.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.880 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:06.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:06.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:00:06.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:00:06.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:00:06.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:00:06.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:00:06.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:00:06.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:00:06.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:00:06.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:00:06.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:00:06.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:00:06.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:00:06.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:00:06.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:00:06.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:00:06.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:00:06.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:00:06.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:00:06.885 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.885 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.885 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.886 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.886 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:06.886 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 54.605938875, 2024-08-30T16:00:06.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:00:06.886 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.886 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.886 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.886 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.886 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.886 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:00:06.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.886 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:06.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:06.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:00:06.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:06.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:00:06.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:00:06.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:00:06.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:00:06.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:00:06.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:00:06.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:00:06.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:00:06.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:00:06.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:00:06.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:00:06.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:00:06.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:00:06.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:00:06.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:00:06.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:00:06.890 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.891 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.891 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.891 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.891 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:06.891 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 54.605181426999998, 2024-08-30T16:00:06.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:00:06.891 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.891 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.892 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:00:06.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.892 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:06.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:06.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:00:06.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:00:06.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:00:06.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:00:06.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:00:06.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:00:06.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:00:06.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:00:06.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:00:06.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:00:06.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:00:06.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:00:06.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:00:06.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:00:06.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:00:06.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:00:06.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:00:06.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:00:06.896 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.896 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.896 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.896 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.896 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:06.896 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 54.604448976999997, 2024-08-30T16:00:06.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:00:06.897 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.897 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.897 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.897 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.897 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.897 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:00:06.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.897 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:06.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:06.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:00:06.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:06.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:00:06.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:00:06.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:00:06.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:00:06.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:00:06.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:00:06.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:00:06.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:00:06.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:00:06.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:00:06.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:00:06.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:00:06.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:00:06.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:00:06.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:00:06.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:00:06.901 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.901 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.901 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.902 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.902 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:06.902 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 54.601910177999997, 2024-08-30T16:00:06.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:00:06.902 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.902 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.902 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.902 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.902 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.902 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:00:06.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.902 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:06.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:06.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:00:06.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:06.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:00:06.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:00:06.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:00:06.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:00:06.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:00:06.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:00:06.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:00:06.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:00:06.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:00:06.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:00:06.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:00:06.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:00:06.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:00:06.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:00:06.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:00:06.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:00:06.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.907 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.907 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.907 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:06.907 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 54.601162176999999, 2024-08-30T16:00:06.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:00:06.907 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.908 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.908 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.908 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.908 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.908 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:00:06.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.908 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:06.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:06.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:00:06.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:00:06.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:00:06.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:00:06.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:00:06.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:00:06.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:00:06.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:00:06.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:00:06.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:00:06.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:00:06.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:00:06.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:00:06.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:00:06.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:00:06.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:00:06.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:00:06.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:00:06.912 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.912 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.912 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.913 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.913 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:06.913 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.829045704999999, 2024-08-30T16:00:06.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:00:06.913 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.913 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.913 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.913 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.913 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.913 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:00:06.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.913 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:06.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:06.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:00:06.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:00:06.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:00:06.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:00:06.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:00:06.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:00:06.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:00:06.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:00:06.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:00:06.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:00:06.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:00:06.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:00:06.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:00:06.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:00:06.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:00:06.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:00:06.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:00:06.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:00:06.918 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.918 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.918 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.918 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.918 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:06.918 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.826988559, 2024-08-30T16:00:06.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:00:06.918 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.918 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.918 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.918 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.919 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.919 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:00:06.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.919 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:06.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:06.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:00:06.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:06.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:00:06.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:00:06.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:00:06.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:00:06.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:00:06.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:00:06.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:00:06.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:00:06.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:00:06.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:00:06.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:00:06.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:00:06.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:00:06.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:00:06.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:00:06.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:00:06.923 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.923 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.923 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.923 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.923 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:06.923 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.564044058999997, 2024-08-30T16:00:06.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:00:06.924 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.924 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.924 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.924 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.924 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.924 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:00:06.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.924 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:06.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:06.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:00:06.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:00:06.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:00:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:00:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:00:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:00:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:00:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:00:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:00:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:00:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:00:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:00:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:00:06.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:00:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:00:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:00:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:00:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:00:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.928 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.562952555999999, 2024-08-30T16:00:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:00:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:00:06.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:00:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:00:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:00:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:00:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:00:06.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:00:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:00:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:00:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:00:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:00:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:00:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:00:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:00:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:00:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:00:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:00:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:00:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.562202970999998, 2024-08-30T16:00:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:00:06.934 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:00:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:00:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:00:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:00:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:00:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:00:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:00:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:00:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:00:06.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:00:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:00:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:00:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:00:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:00:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:00:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:00:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:00:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:00:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.939 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.528471439, 2024-08-30T16:00:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:00:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:00:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:00:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:00:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:00:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:00:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:00:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:00:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:00:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:00:06.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:00:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:00:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:00:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:00:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:00:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:00:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:00:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:00:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:00:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:00:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.944 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.527945430999999, 2024-08-30T16:00:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:00:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:00:06.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:00:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:00:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:00:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:00:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:00:06.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:00:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:00:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:00:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:00:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:00:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:00:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:00:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:00:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:00:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:00:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:00:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:00:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.527489868000004, 2024-08-30T16:00:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:00:06.950 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:00:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:00:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:00:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:00:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:00:06.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:00:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:00:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:00:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:00:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:00:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:00:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:00:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:00:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:00:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:00:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:00:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:00:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:00:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:00:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.955 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:00:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.856974551999997, 2024-08-30T16:00:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:00:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:06.956 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:00:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:00:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:00:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:00:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:00:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:00:06.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:00:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:00:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:00:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:00:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:00:06.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:00:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:00:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:00:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:00:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:00:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:06.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:00:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:00:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:06.961 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:06.961 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:06.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:00:06.963 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:00:06.963 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:06.961+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:07.064 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:07.062+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:07.144 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:07.145 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:07.145 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:07.145 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:07.153 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:00:07.164 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:07.163+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:07.265 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:07.264+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:07.321 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:07.321 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:07.321 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:07.322 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:07.322 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:07.331 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:00:07.375 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:07.373+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:07.467 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:07.466+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:07.471 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:07.471 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:07.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:07.471 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:07.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.471 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:00:07.471 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:07.471 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.931216034999998, 2024-08-30T16:00:07.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:00:07.471 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:07.471 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:07.471 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:07.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:07.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:07.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:07.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:07.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:07.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:07.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:07.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:00:07.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:00:07.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:07.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:00:07.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:00:07.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:00:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:00:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:00:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:00:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:00:07.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:00:07.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:07.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:00:07.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:00:07.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:07.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:00:07.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:00:07.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:07.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:00:07.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:00:07.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:07.476 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:07.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:07.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.476 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:00:07.476 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:07.476 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 39.812767889, 2024-08-30T16:00:07.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:00:07.476 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:07.476 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:07.477 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:07.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:07.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:07.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:07.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:07.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:07.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:07.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:07.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:00:07.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:00:07.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:07.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:00:07.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:00:07.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:07.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:00:07.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:00:07.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:07.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:00:07.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:00:07.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:07.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:00:07.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:00:07.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:07.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:00:07.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:00:07.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:07.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:00:07.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:00:07.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:07.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:07.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:07.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:00:07.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:00:07.481 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:07.481 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:07.481 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:07.481 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:07.481 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:07.481 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:07.482 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:00:07.484 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:07.949 30228== Warning: unimplemented fcntl command: 1036 2024-08-30T16:00:07.568 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:07.566+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:07.625 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:00:07.635 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":8,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:00:07.644 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.7 to 1 (10000) 2024-08-30T16:00:07.667 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-08-30T16:00:07.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:00:07.677 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:07.677 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:07.677 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:07.677 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:07.687 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:00:07.708 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:07.707+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:07.770 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:07.769+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:07.860 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:07.860 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:07.860 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:07.860 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:07.860 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:07.869 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:00:07.871 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:07.870+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:07.892 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:00:07.892 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:00:07.972 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:07.970+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:07.986 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:00:07.986 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:00:08.007 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:08.007 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 39.618369029999997, 2024-08-30T16:00:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:00:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:00:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:00:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:00:08.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:08.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:00:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:00:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:00:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:00:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:00:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:00:08.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:00:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:00:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:00:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:00:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:08.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:00:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:00:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:08.012 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:08.012 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:08.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:00:08.073 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:08.072+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:08.091 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:00:08.092 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:00:08.152 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:08.152 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:08.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:00:08.207 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:00:08.207 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:00:08.207 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:08.205+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:08.275 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:08.274+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:08.290 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:00:08.290 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:00:08.293 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:08.293 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:08.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:00:08.353 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:00:08.353 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:00:08.363 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-08-30T16:00:07.877624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218564,"kb_used_data":2380,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756156,"statfs":{"total":773094113280,"available":772870303744,"internally_reserved":0,"allocated":2437120,"data_stored":1697550,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":72,"apply_latency_ms":72,"commit_latency_ns":72000000,"apply_latency_ns":72000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000582"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-08-30T16:00:02.393497+0000","last_change":"2024-08-30T15:59:27.646756+0000","last_active":"2024-08-30T16:00:02.393497+0000","last_peered":"2024-08-30T16:00:02.393497+0000","last_clean":"2024-08-30T16:00:02.393497+0000","last_became_active":"2024-08-30T15:59:25.342321+0000","last_became_peered":"2024-08-30T15:59:25.342321+0000","last_unstale":"2024-08-30T16:00:02.393497+0000","last_undegraded":"2024-08-30T16:00:02.393497+0000","last_fullsized":"2024-08-30T16:00:02.393497+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:53:42.927368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.116914037,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-08-30T16:00:05.446708+0000","last_change":"2024-08-30T15:59:28.156737+0000","last_active":"2024-08-30T16:00:05.446708+0000","last_peered":"2024-08-30T16:00:05.446708+0000","last_clean":"2024-08-30T16:00:05.446708+0000","last_became_active":"2024-08-30T15:59:25.662524+0000","last_became_peered":"2024-08-30T15:59:25.662524+0000","last_unstale":"2024-08-30T16:00:05.446708+0000","last_undegraded":"2024-08-30T16:00:05.446708+0000","last_fullsized":"2024-08-30T16:00:05.446708+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:05:30.092449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064329163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-08-30T16:00:02.389883+0000","last_change":"2024-08-30T15:59:27.633478+0000","last_active":"2024-08-30T16:00:02.389883+0000","last_peered":"2024-08-30T16:00:02.389883+0000","last_clean":"2024-08-30T16:00:02.389883+0000","last_became_active":"2024-08-30T15:59:24.572730+0000","last_became_peered":"2024-08-30T15:59:24.572730+0000","last_unstale":"2024-08-30T16:00:02.389883+0000","last_undegraded":"2024-08-30T16:00:02.389883+0000","last_fullsized":"2024-08-30T16:00:02.389883+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:36:20.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-08-30T16:00:02.117011+0000","last_change":"2024-08-30T15:59:27.584250+0000","last_active":"2024-08-30T16:00:02.117011+0000","last_peered":"2024-08-30T16:00:02.117011+0000","last_clean":"2024-08-30T16:00:02.117011+0000","last_became_active":"2024-08-30T15:59:24.687233+0000","last_became_peered":"2024-08-30T15:59:24.687233+0000","last_unstale":"2024-08-30T16:00:02.117011+0000","last_undegraded":"2024-08-30T16:00:02.117011+0000","last_fullsized":"2024-08-30T16:00:02.117011+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:49:09.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":25,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-08-30T16:00:02.381787+0000","last_change":"2024-08-30T15:59:27.623039+0000","last_active":"2024-08-30T16:00:02.381787+0000","last_peered":"2024-08-30T16:00:02.381787+0000","last_clean":"2024-08-30T16:00:02.381787+0000","last_became_active":"2024-08-30T15:59:24.688486+0000","last_became_peered":"2024-08-30T15:59:24.688486+0000","last_unstale":"2024-08-30T16:00:02.381787+0000","last_undegraded":"2024-08-30T16:00:02.381787+0000","last_fullsized":"2024-08-30T16:00:02.381787+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:22:51.391036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-30T16:00:01.442846+0000","last_change":"2024-08-30T15:59:27.825532+0000","last_active":"2024-08-30T16:00:01.442846+0000","last_peered":"2024-08-30T16:00:01.442846+0000","last_clean":"2024-08-30T16:00:01.442846+0000","last_became_active":"2024-08-30T15:59:25.469390+0000","last_became_peered":"2024-08-30T15:59:25.469390+0000","last_unstale":"2024-08-30T16:00:01.442846+0000","last_undegraded":"2024-08-30T16:00:01.442846+0000","last_fullsized":"2024-08-30T16:00:01.442846+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T17:46:39.688911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-08-30T16:00:02.379886+0000","last_change":"2024-08-30T15:59:27.583986+0000","last_active":"2024-08-30T16:00:02.379886+0000","last_peered":"2024-08-30T16:00:02.379886+0000","last_clean":"2024-08-30T16:00:02.379886+0000","last_became_active":"2024-08-30T15:59:24.528536+0000","last_became_peered":"2024-08-30T15:59:24.528536+0000","last_unstale":"2024-08-30T16:00:02.379886+0000","last_undegraded":"2024-08-30T16:00:02.379886+0000","last_fullsized":"2024-08-30T16:00:02.379886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:18:40.602710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":24,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-08-30T16:00:02.391655+0000","last_change":"2024-08-30T15:59:27.636613+0000","last_active":"2024-08-30T16:00:02.391655+0000","last_peered":"2024-08-30T16:00:02.391655+0000","last_clean":"2024-08-30T16:00:02.391655+0000","last_became_active":"2024-08-30T15:59:25.021268+0000","last_became_peered":"2024-08-30T15:59:25.021268+0000","last_unstale":"2024-08-30T16:00:02.391655+0000","last_undegraded":"2024-08-30T16:00:02.391655+0000","last_fullsized":"2024-08-30T16:00:02.391655+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T00:56:28.767086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113256018,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-30T16:00:01.883967+0000","last_change":"2024-08-30T15:59:21.623783+0000","last_active":"2024-08-30T16:00:01.883967+0000","last_peered":"2024-08-30T16:00:01.883967+0000","last_clean":"2024-08-30T16:00:01.883967+0000","last_became_active":"2024-08-30T15:59:20.848924+0000","last_became_peered":"2024-08-30T15:59:20.848924+0000","last_unstale":"2024-08-30T16:00:01.883967+0000","last_undegraded":"2024-08-30T16:00:01.883967+0000","last_fullsized":"2024-08-30T16:00:01.883967+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:14:52.575421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":795680,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280530,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":508420,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8320000000000007}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.962999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.164999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.550000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.431000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.946}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.786999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313233,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":101984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.9880000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.531000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.792999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.472}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.94}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.101000000000001}]}]},{"osd":6,"up_from":21,"seq":90194313232,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":106301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.460000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.537000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6560000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.343}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.962}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.620999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.994}]}]},{"osd":3,"up_from":21,"seq":90194313232,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":110599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.965}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.216000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.643999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.619}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.815000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.047999999999998}]}]},{"osd":4,"up_from":20,"seq":85899345937,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":119176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1289999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.818}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.459}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.161999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.032}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.675000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1839999999999993}]}]},{"osd":2,"up_from":16,"seq":68719476751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":315392,"data_stored":123474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9480000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.654}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.34}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.364000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.937999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.004999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509454,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":119176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.352}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.045000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":36.715000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":56.200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.265000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":34.569000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":50.229999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":327680,"data_stored":508420,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.835}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5589999999999993}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.431999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.564}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.922999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.454999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":43.604999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:00:08.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:00:08.375 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:08.374+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:08.376 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:00:08.376 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:00:08.458 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:00:08.459 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:00:08.476 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:08.476 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:08.483 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:08.481+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:08.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:00:08.543 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:00:08.577 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:08.576+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:08.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-08-30T16:00:08.616 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:08.616 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:08.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:08.710 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:08.709+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:08.779 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:08.778+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:08.805 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:08.805 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:08.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:08.880 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:08.879+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:08.932 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:00:08.933 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:00:08.949 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-08-30T16:00:07.877624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218564,"kb_used_data":2380,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756156,"statfs":{"total":773094113280,"available":772870303744,"internally_reserved":0,"allocated":2437120,"data_stored":1697550,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":72,"apply_latency_ms":72,"commit_latency_ns":72000000,"apply_latency_ns":72000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000582"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-08-30T16:00:02.393497+0000","last_change":"2024-08-30T15:59:27.646756+0000","last_active":"2024-08-30T16:00:02.393497+0000","last_peered":"2024-08-30T16:00:02.393497+0000","last_clean":"2024-08-30T16:00:02.393497+0000","last_became_active":"2024-08-30T15:59:25.342321+0000","last_became_peered":"2024-08-30T15:59:25.342321+0000","last_unstale":"2024-08-30T16:00:02.393497+0000","last_undegraded":"2024-08-30T16:00:02.393497+0000","last_fullsized":"2024-08-30T16:00:02.393497+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:53:42.927368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.116914037,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-08-30T16:00:05.446708+0000","last_change":"2024-08-30T15:59:28.156737+0000","last_active":"2024-08-30T16:00:05.446708+0000","last_peered":"2024-08-30T16:00:05.446708+0000","last_clean":"2024-08-30T16:00:05.446708+0000","last_became_active":"2024-08-30T15:59:25.662524+0000","last_became_peered":"2024-08-30T15:59:25.662524+0000","last_unstale":"2024-08-30T16:00:05.446708+0000","last_undegraded":"2024-08-30T16:00:05.446708+0000","last_fullsized":"2024-08-30T16:00:05.446708+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:05:30.092449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064329163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-08-30T16:00:02.389883+0000","last_change":"2024-08-30T15:59:27.633478+0000","last_active":"2024-08-30T16:00:02.389883+0000","last_peered":"2024-08-30T16:00:02.389883+0000","last_clean":"2024-08-30T16:00:02.389883+0000","last_became_active":"2024-08-30T15:59:24.572730+0000","last_became_peered":"2024-08-30T15:59:24.572730+0000","last_unstale":"2024-08-30T16:00:02.389883+0000","last_undegraded":"2024-08-30T16:00:02.389883+0000","last_fullsized":"2024-08-30T16:00:02.389883+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:36:20.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-08-30T16:00:02.117011+0000","last_change":"2024-08-30T15:59:27.584250+0000","last_active":"2024-08-30T16:00:02.117011+0000","last_peered":"2024-08-30T16:00:02.117011+0000","last_clean":"2024-08-30T16:00:02.117011+0000","last_became_active":"2024-08-30T15:59:24.687233+0000","last_became_peered":"2024-08-30T15:59:24.687233+0000","last_unstale":"2024-08-30T16:00:02.117011+0000","last_undegraded":"2024-08-30T16:00:02.117011+0000","last_fullsized":"2024-08-30T16:00:02.117011+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:49:09.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":25,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-08-30T16:00:02.381787+0000","last_change":"2024-08-30T15:59:27.623039+0000","last_active":"2024-08-30T16:00:02.381787+0000","last_peered":"2024-08-30T16:00:02.381787+0000","last_clean":"2024-08-30T16:00:02.381787+0000","last_became_active":"2024-08-30T15:59:24.688486+0000","last_became_peered":"2024-08-30T15:59:24.688486+0000","last_unstale":"2024-08-30T16:00:02.381787+0000","last_undegraded":"2024-08-30T16:00:02.381787+0000","last_fullsized":"2024-08-30T16:00:02.381787+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:22:51.391036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-30T16:00:01.442846+0000","last_change":"2024-08-30T15:59:27.825532+0000","last_active":"2024-08-30T16:00:01.442846+0000","last_peered":"2024-08-30T16:00:01.442846+0000","last_clean":"2024-08-30T16:00:01.442846+0000","last_became_active":"2024-08-30T15:59:25.469390+0000","last_became_peered":"2024-08-30T15:59:25.469390+0000","last_unstale":"2024-08-30T16:00:01.442846+0000","last_undegraded":"2024-08-30T16:00:01.442846+0000","last_fullsized":"2024-08-30T16:00:01.442846+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T17:46:39.688911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-08-30T16:00:02.379886+0000","last_change":"2024-08-30T15:59:27.583986+0000","last_active":"2024-08-30T16:00:02.379886+0000","last_peered":"2024-08-30T16:00:02.379886+0000","last_clean":"2024-08-30T16:00:02.379886+0000","last_became_active":"2024-08-30T15:59:24.528536+0000","last_became_peered":"2024-08-30T15:59:24.528536+0000","last_unstale":"2024-08-30T16:00:02.379886+0000","last_undegraded":"2024-08-30T16:00:02.379886+0000","last_fullsized":"2024-08-30T16:00:02.379886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:18:40.602710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":24,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-08-30T16:00:02.391655+0000","last_change":"2024-08-30T15:59:27.636613+0000","last_active":"2024-08-30T16:00:02.391655+0000","last_peered":"2024-08-30T16:00:02.391655+0000","last_clean":"2024-08-30T16:00:02.391655+0000","last_became_active":"2024-08-30T15:59:25.021268+0000","last_became_peered":"2024-08-30T15:59:25.021268+0000","last_unstale":"2024-08-30T16:00:02.391655+0000","last_undegraded":"2024-08-30T16:00:02.391655+0000","last_fullsized":"2024-08-30T16:00:02.391655+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T00:56:28.767086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113256018,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-30T16:00:01.883967+0000","last_change":"2024-08-30T15:59:21.623783+0000","last_active":"2024-08-30T16:00:01.883967+0000","last_peered":"2024-08-30T16:00:01.883967+0000","last_clean":"2024-08-30T16:00:01.883967+0000","last_became_active":"2024-08-30T15:59:20.848924+0000","last_became_peered":"2024-08-30T15:59:20.848924+0000","last_unstale":"2024-08-30T16:00:01.883967+0000","last_undegraded":"2024-08-30T16:00:01.883967+0000","last_fullsized":"2024-08-30T16:00:01.883967+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:14:52.575421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":795680,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280530,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":508420,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8320000000000007}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.962999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.164999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.550000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.431000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.946}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.786999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313233,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":101984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.9880000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.531000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.792999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.472}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.94}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.101000000000001}]}]},{"osd":6,"up_from":21,"seq":90194313232,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":106301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.460000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.537000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6560000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.343}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.962}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.620999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.994}]}]},{"osd":3,"up_from":21,"seq":90194313232,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":110599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.965}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.216000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.643999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.619}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.815000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.047999999999998}]}]},{"osd":4,"up_from":20,"seq":85899345937,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":119176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1289999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.818}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.459}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.161999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.032}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.675000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1839999999999993}]}]},{"osd":2,"up_from":16,"seq":68719476751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":315392,"data_stored":123474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9480000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.654}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.34}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.364000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.937999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.004999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509454,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":119176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.352}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.045000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":36.715000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":56.200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.265000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":34.569000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":50.229999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":327680,"data_stored":508420,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.835}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5589999999999993}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.431999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.564}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.922999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.454999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":43.604999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:00:08.950 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-08-30T16:00:08.950 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:00:08.950 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-30T16:00:08.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-08-30T16:00:08.981 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:08.980+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:09.005 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:09.005 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:09.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:09.082 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:09.081+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:09.163 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:09.163 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:09.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:09.182 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:09.181+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:09.283 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:09.281+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:09.344 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:09.344 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:09.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:09.385 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:09.383+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:09.485 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:09.485 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:09.485 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:09.484+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:09.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:09.585 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:00:09.586 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:09.586 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 38, 2024-08-30T16:00:09.586 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "9d72e2e2-0554-42af-8e62-9b1a3a804112", 2024-08-30T16:00:09.586 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-08-30T15:58:20.129410+0000", 2024-08-30T16:00:09.586 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-08-30T16:00:07.533590+0000", 2024-08-30T16:00:09.586 INFO:teuthology.orchestra.run.smithi050.stdout: "last_up_change": "2024-08-30T15:59:19.280372+0000", 2024-08-30T16:00:09.586 INFO:teuthology.orchestra.run.smithi050.stdout: "last_in_change": "2024-08-30T15:58:25.430425+0000", 2024-08-30T16:00:09.586 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T16:00:09.586 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_num": 5799936, 2024-08-30T16:00:09.586 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_set": [ 2024-08-30T16:00:09.586 INFO:teuthology.orchestra.run.smithi050.stdout: "pglog_hardlimit", 2024-08-30T16:00:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: "purged_snapdirs", 2024-08-30T16:00:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_deletes", 2024-08-30T16:00:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: "sortbitwise" 2024-08-30T16:00:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:00:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_version": 7, 2024-08-30T16:00:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T16:00:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T16:00:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T16:00:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_snapshot": "", 2024-08-30T16:00:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_max": 2, 2024-08-30T16:00:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd": 8, 2024-08-30T16:00:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: "require_min_compat_client": "luminous", 2024-08-30T16:00:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: "min_compat_client": "jewel", 2024-08-30T16:00:09.587 INFO:teuthology.orchestra.run.smithi050.stdout: "require_osd_release": "reef", 2024-08-30T16:00:09.588 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_crimson": false, 2024-08-30T16:00:09.588 INFO:teuthology.orchestra.run.smithi050.stdout: "pools": [ 2024-08-30T16:00:09.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.588 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 1, 2024-08-30T16:00:09.588 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".mgr", 2024-08-30T16:00:09.588 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-08-30T15:59:08.355540+0000", 2024-08-30T16:00:09.588 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-08-30T16:00:09.588 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-08-30T16:00:09.588 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-08-30T16:00:09.588 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 2, 2024-08-30T16:00:09.588 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 1, 2024-08-30T16:00:09.588 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-08-30T16:00:09.588 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-08-30T16:00:09.588 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-08-30T16:00:09.589 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T16:00:09.589 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T16:00:09.589 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-08-30T16:00:09.589 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-08-30T16:00:09.589 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 1, 2024-08-30T16:00:09.589 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 1, 2024-08-30T16:00:09.589 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 1, 2024-08-30T16:00:09.589 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 1, 2024-08-30T16:00:09.589 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 1, 2024-08-30T16:00:09.589 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-08-30T16:00:09.589 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-08-30T16:00:09.589 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-08-30T16:00:09.589 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-08-30T16:00:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-08-30T16:00:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-08-30T16:00:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-08-30T16:00:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "19", 2024-08-30T16:00:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-08-30T16:00:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T16:00:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T16:00:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-08-30T16:00:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-08-30T16:00:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-08-30T16:00:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-08-30T16:00:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-08-30T16:00:09.590 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-08-30T16:00:09.591 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-08-30T16:00:09.591 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-08-30T16:00:09.591 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-08-30T16:00:09.591 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-08-30T16:00:09.591 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-08-30T16:00:09.591 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-08-30T16:00:09.591 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-08-30T16:00:09.591 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-08-30T16:00:09.591 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-08-30T16:00:09.591 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T16:00:09.591 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T16:00:09.591 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T16:00:09.591 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-08-30T16:00:09.591 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-08-30T16:00:09.592 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-08-30T16:00:09.592 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-08-30T16:00:09.592 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-08-30T16:00:09.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.592 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-08-30T16:00:09.592 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-08-30T16:00:09.592 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-08-30T16:00:09.592 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-08-30T16:00:09.592 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-08-30T16:00:09.592 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T16:00:09.592 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-08-30T16:00:09.592 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-08-30T16:00:09.592 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-08-30T16:00:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-08-30T16:00:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-08-30T16:00:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-08-30T16:00:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_max": 32, 2024-08-30T16:00:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_min": 1 2024-08-30T16:00:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-08-30T16:00:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": {} 2024-08-30T16:00:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-08-30T16:00:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 8, 2024-08-30T16:00:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 8, 2024-08-30T16:00:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.25, 2024-08-30T16:00:09.593 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-08-30T16:00:09.594 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-08-30T16:00:09.594 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-08-30T16:00:09.594 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-08-30T16:00:09.594 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-08-30T16:00:09.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.594 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-08-30T16:00:09.594 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "rbd", 2024-08-30T16:00:09.594 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-08-30T15:59:21.696796+0000", 2024-08-30T16:00:09.594 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-08-30T16:00:09.594 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-08-30T16:00:09.594 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-08-30T16:00:09.594 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 2, 2024-08-30T16:00:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 1, 2024-08-30T16:00:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-08-30T16:00:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-08-30T16:00:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-08-30T16:00:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T16:00:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T16:00:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-08-30T16:00:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-08-30T16:00:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 8, 2024-08-30T16:00:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 8, 2024-08-30T16:00:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 8, 2024-08-30T16:00:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 8, 2024-08-30T16:00:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 8, 2024-08-30T16:00:09.595 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-08-30T16:00:09.596 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-08-30T16:00:09.596 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-08-30T16:00:09.596 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-08-30T16:00:09.596 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-08-30T16:00:09.596 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-08-30T16:00:09.596 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-08-30T16:00:09.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.596 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "28", 2024-08-30T16:00:09.596 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-08-30T16:00:09.596 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T16:00:09.596 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T16:00:09.596 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-08-30T16:00:09.596 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-08-30T16:00:09.597 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 3, 2024-08-30T16:00:09.597 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 28, 2024-08-30T16:00:09.597 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-08-30T16:00:09.597 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-08-30T16:00:09.597 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-08-30T16:00:09.597 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-08-30T16:00:09.597 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-08-30T16:00:09.597 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-08-30T16:00:09.597 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-08-30T16:00:09.597 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-08-30T16:00:09.597 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-08-30T16:00:09.597 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-08-30T16:00:09.597 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-08-30T16:00:09.597 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T16:00:09.598 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T16:00:09.598 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T16:00:09.598 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-08-30T16:00:09.598 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-08-30T16:00:09.598 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-08-30T16:00:09.598 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-08-30T16:00:09.598 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-08-30T16:00:09.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.598 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-08-30T16:00:09.598 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-08-30T16:00:09.598 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-08-30T16:00:09.598 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-08-30T16:00:09.598 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-08-30T16:00:09.598 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T16:00:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-08-30T16:00:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-08-30T16:00:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-08-30T16:00:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-08-30T16:00:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-08-30T16:00:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-08-30T16:00:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-08-30T16:00:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd": {} 2024-08-30T16:00:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-08-30T16:00:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 4.9800000190734863, 2024-08-30T16:00:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 4.9800000190734863, 2024-08-30T16:00:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.87999999523162842, 2024-08-30T16:00:09.599 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 4.380000114440918, 2024-08-30T16:00:09.600 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 4.380000114440918, 2024-08-30T16:00:09.600 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-08-30T16:00:09.600 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-08-30T16:00:09.600 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-08-30T16:00:09.600 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.600 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.600 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:00:09.600 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-08-30T16:00:09.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.600 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-08-30T16:00:09.600 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4", 2024-08-30T16:00:09.601 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:00:09.601 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:00:09.601 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:00:09.601 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:00:09.601 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:00:09.601 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:00:09.601 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 15, 2024-08-30T16:00:09.601 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:00:09.601 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:00:09.601 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:00:09.601 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:00:09.601 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.601 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6812", 2024-08-30T16:00:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:00:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:00:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6813", 2024-08-30T16:00:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:00:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.603 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:00:09.603 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.603 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.603 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6815", 2024-08-30T16:00:09.603 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:00:09.603 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.603 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.604 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:00:09.604 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.604 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.604 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6814", 2024-08-30T16:00:09.604 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:00:09.604 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.604 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.604 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6812/1957399494", 2024-08-30T16:00:09.604 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6813/1957399494", 2024-08-30T16:00:09.604 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6815/1957399494", 2024-08-30T16:00:09.604 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6814/1957399494", 2024-08-30T16:00:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:00:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:00:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:00:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-08-30T16:00:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "0fe11a1c-4326-4bfd-a2a4-bff14023f301", 2024-08-30T16:00:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:00:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:00:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:00:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:00:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:00:09.605 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:00:09.606 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 22, 2024-08-30T16:00:09.606 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 22, 2024-08-30T16:00:09.606 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:00:09.606 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:00:09.606 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:00:09.606 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.606 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.606 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6808", 2024-08-30T16:00:09.606 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:00:09.606 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.606 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.606 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:00:09.607 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.607 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.607 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6809", 2024-08-30T16:00:09.607 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:00:09.607 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.607 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.607 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:00:09.607 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.607 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.607 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6811", 2024-08-30T16:00:09.608 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:00:09.608 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.608 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.608 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:00:09.608 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.608 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.608 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6810", 2024-08-30T16:00:09.608 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:00:09.608 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.608 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.609 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6808/3208178756", 2024-08-30T16:00:09.609 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6809/3208178756", 2024-08-30T16:00:09.609 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6811/3208178756", 2024-08-30T16:00:09.609 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6810/3208178756", 2024-08-30T16:00:09.609 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:00:09.609 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:00:09.609 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:00:09.609 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.609 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-08-30T16:00:09.609 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "a9d5354b-3600-41ed-80c5-a3439da7aca6", 2024-08-30T16:00:09.609 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:00:09.610 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:00:09.610 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:00:09.610 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:00:09.610 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:00:09.610 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:00:09.610 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 16, 2024-08-30T16:00:09.610 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:00:09.610 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:00:09.610 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:00:09.610 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:00:09.610 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.610 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.611 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6804", 2024-08-30T16:00:09.611 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:00:09.611 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.611 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.611 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:00:09.611 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.611 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.611 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6805", 2024-08-30T16:00:09.611 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:00:09.611 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.611 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.612 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:00:09.612 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.612 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.612 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6807", 2024-08-30T16:00:09.612 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:00:09.612 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.612 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.612 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:00:09.612 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.613 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.613 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6806", 2024-08-30T16:00:09.613 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:00:09.613 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.613 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.613 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6804/287018800", 2024-08-30T16:00:09.613 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6805/287018800", 2024-08-30T16:00:09.613 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6807/287018800", 2024-08-30T16:00:09.613 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6806/287018800", 2024-08-30T16:00:09.613 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:00:09.613 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:00:09.614 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:00:09.614 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.614 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-08-30T16:00:09.615 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "78bcc398-a7e4-4efc-9836-1dfb94407c9c", 2024-08-30T16:00:09.615 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:00:09.615 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:00:09.615 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:00:09.615 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:00:09.615 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:00:09.615 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:00:09.615 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:00:09.615 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 24, 2024-08-30T16:00:09.615 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:00:09.615 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:00:09.615 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:00:09.615 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.616 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.616 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6800", 2024-08-30T16:00:09.616 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:00:09.616 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.616 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.616 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:00:09.616 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.616 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.616 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6801", 2024-08-30T16:00:09.616 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:00:09.617 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.617 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.617 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:00:09.617 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.617 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.617 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6803", 2024-08-30T16:00:09.617 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:00:09.617 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.617 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:00:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6802", 2024-08-30T16:00:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:00:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6800/1140717714", 2024-08-30T16:00:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6801/1140717714", 2024-08-30T16:00:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6803/1140717714", 2024-08-30T16:00:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6802/1140717714", 2024-08-30T16:00:09.618 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:00:09.619 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:00:09.619 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:00:09.619 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.619 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-08-30T16:00:09.619 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "53a1060b-7985-4c2c-820e-64b158056113", 2024-08-30T16:00:09.619 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:00:09.619 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:00:09.619 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:00:09.619 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:00:09.619 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:00:09.619 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:00:09.620 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 20, 2024-08-30T16:00:09.620 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 24, 2024-08-30T16:00:09.620 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:00:09.620 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:00:09.620 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:00:09.620 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.620 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.620 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6805", 2024-08-30T16:00:09.620 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:00:09.620 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.620 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.620 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:00:09.621 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.621 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.621 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6806", 2024-08-30T16:00:09.621 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:00:09.621 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.621 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.621 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:00:09.621 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.621 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.622 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6808", 2024-08-30T16:00:09.622 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:00:09.622 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.622 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.622 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:00:09.622 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.622 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.622 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6807", 2024-08-30T16:00:09.622 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:00:09.622 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.622 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.623 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6805/1468073345", 2024-08-30T16:00:09.623 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6806/1468073345", 2024-08-30T16:00:09.623 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6808/1468073345", 2024-08-30T16:00:09.623 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6807/1468073345", 2024-08-30T16:00:09.623 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:00:09.623 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:00:09.623 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:00:09.623 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.623 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-08-30T16:00:09.623 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "853a03f7-227d-4f98-9e5c-c29896820746", 2024-08-30T16:00:09.624 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:00:09.624 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:00:09.624 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:00:09.624 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:00:09.624 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:00:09.624 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:00:09.624 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 13, 2024-08-30T16:00:09.624 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 24, 2024-08-30T16:00:09.624 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:00:09.624 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:00:09.624 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:00:09.624 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6809", 2024-08-30T16:00:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:00:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:00:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6810", 2024-08-30T16:00:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:00:09.625 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.626 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.626 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:00:09.626 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.626 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.626 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6812", 2024-08-30T16:00:09.626 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:00:09.626 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.626 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.626 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:00:09.627 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.627 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.627 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6811", 2024-08-30T16:00:09.627 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:00:09.627 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.627 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.627 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6809/3876025616", 2024-08-30T16:00:09.627 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6810/3876025616", 2024-08-30T16:00:09.627 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6812/3876025616", 2024-08-30T16:00:09.627 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6811/3876025616", 2024-08-30T16:00:09.627 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:00:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:00:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:00:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-08-30T16:00:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "bc978060-bf28-4027-8bba-7d6d8659bfc5", 2024-08-30T16:00:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:00:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:00:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:00:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:00:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:00:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:00:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:00:09.629 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:00:09.629 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:00:09.629 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:00:09.629 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:00:09.629 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.629 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.629 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6801", 2024-08-30T16:00:09.629 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:00:09.629 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.629 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.629 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:00:09.630 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.630 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.630 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6802", 2024-08-30T16:00:09.630 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:00:09.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.630 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.630 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:00:09.630 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.630 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.630 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6804", 2024-08-30T16:00:09.631 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:00:09.631 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.631 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.631 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:00:09.631 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.631 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.631 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6803", 2024-08-30T16:00:09.631 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:00:09.631 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.631 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.632 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6801/128519122", 2024-08-30T16:00:09.632 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6802/128519122", 2024-08-30T16:00:09.632 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6804/128519122", 2024-08-30T16:00:09.632 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6803/128519122", 2024-08-30T16:00:09.632 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:00:09.632 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:00:09.632 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:00:09.632 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.632 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-08-30T16:00:09.632 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "45df5c6f-b124-46d1-9efb-62b78bc584ed", 2024-08-30T16:00:09.632 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:00:09.633 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:00:09.633 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:00:09.633 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:00:09.633 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:00:09.633 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:00:09.633 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:00:09.633 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 24, 2024-08-30T16:00:09.633 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:00:09.633 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:00:09.633 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:00:09.633 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.633 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.634 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6813", 2024-08-30T16:00:09.634 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:00:09.634 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.634 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.634 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:00:09.634 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.634 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.634 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6814", 2024-08-30T16:00:09.634 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:00:09.634 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.634 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.635 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:00:09.635 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.635 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.635 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6816", 2024-08-30T16:00:09.635 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:00:09.635 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.635 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.635 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:00:09.635 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:00:09.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.636 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:00:09.636 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6815", 2024-08-30T16:00:09.636 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:00:09.636 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.636 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.636 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6813/68721277", 2024-08-30T16:00:09.636 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6814/68721277", 2024-08-30T16:00:09.636 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6816/68721277", 2024-08-30T16:00:09.636 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6815/68721277", 2024-08-30T16:00:09.636 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:00:09.637 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:00:09.637 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:00:09.637 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:09.637 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.637 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:00:09.637 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_xinfo": [ 2024-08-30T16:00:09.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.637 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-08-30T16:00:09.637 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:00:09.637 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:00:09.637 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:00:09.637 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:00:09.637 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:00:09.638 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.911895+0000", 2024-08-30T16:00:09.638 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:00:09.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.638 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-08-30T16:00:09.638 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:00:09.638 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:00:09.638 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:00:09.638 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:00:09.638 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:00:09.638 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:04.786543+0000", 2024-08-30T16:00:09.638 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:00:09.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.639 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-08-30T16:00:09.639 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:00:09.639 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:00:09.639 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:00:09.639 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:00:09.639 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:00:09.639 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:04.197952+0000", 2024-08-30T16:00:09.639 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:00:09.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.639 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-08-30T16:00:09.639 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:00:09.640 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:00:09.640 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:00:09.640 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:00:09.640 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:00:09.640 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.680847+0000", 2024-08-30T16:00:09.640 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:00:09.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.640 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-08-30T16:00:09.640 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:00:09.640 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:00:09.640 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:00:09.641 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:00:09.641 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:00:09.641 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:01.937600+0000", 2024-08-30T16:00:09.641 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:00:09.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.641 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-08-30T16:00:09.641 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:00:09.641 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:00:09.641 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:00:09.641 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:00:09.641 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:00:09.641 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:02.082761+0000", 2024-08-30T16:00:09.642 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:00:09.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.642 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-08-30T16:00:09.642 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:00:09.642 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:00:09.642 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:00:09.642 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:00:09.642 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:00:09.642 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.401025+0000", 2024-08-30T16:00:09.642 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:00:09.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:09.643 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-08-30T16:00:09.643 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:00:09.643 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:00:09.643 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:00:09.643 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:00:09.643 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:00:09.643 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.129371+0000", 2024-08-30T16:00:09.643 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:00:09.643 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.643 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:00:09.643 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap": [], 2024-08-30T16:00:09.643 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_items": [], 2024-08-30T16:00:09.644 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_primaries": [], 2024-08-30T16:00:09.644 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_temp": [], 2024-08-30T16:00:09.644 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_temp": [], 2024-08-30T16:00:09.644 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklist": {}, 2024-08-30T16:00:09.644 INFO:teuthology.orchestra.run.smithi050.stdout: "range_blocklist": {}, 2024-08-30T16:00:09.644 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profiles": { 2024-08-30T16:00:09.644 INFO:teuthology.orchestra.run.smithi050.stdout: "default": { 2024-08-30T16:00:09.644 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-08-30T16:00:09.644 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-08-30T16:00:09.644 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-08-30T16:00:09.644 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-08-30T16:00:09.644 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van" 2024-08-30T16:00:09.644 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:09.645 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps_queue": [], 2024-08-30T16:00:09.645 INFO:teuthology.orchestra.run.smithi050.stdout: "new_removed_snaps": [], 2024-08-30T16:00:09.645 INFO:teuthology.orchestra.run.smithi050.stdout: "new_purged_snaps": [], 2024-08-30T16:00:09.645 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_node_flags": {}, 2024-08-30T16:00:09.645 INFO:teuthology.orchestra.run.smithi050.stdout: "device_class_flags": {}, 2024-08-30T16:00:09.645 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": { 2024-08-30T16:00:09.645 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_enabled": false, 2024-08-30T16:00:09.645 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_bucket_count": 0, 2024-08-30T16:00:09.645 INFO:teuthology.orchestra.run.smithi050.stdout: "degraded_stretch_mode": 0, 2024-08-30T16:00:09.645 INFO:teuthology.orchestra.run.smithi050.stdout: "recovering_stretch_mode": 0, 2024-08-30T16:00:09.645 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_bucket": 0 2024-08-30T16:00:09.645 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:09.646 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:09.646 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:09.585+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:09.648 INFO:tasks.thrashosds.thrasher:j is {'epoch': 38, 'fsid': '9d72e2e2-0554-42af-8e62-9b1a3a804112', 'created': '2024-08-30T15:58:20.129410+0000', 'modified': '2024-08-30T16:00:07.533590+0000', 'last_up_change': '2024-08-30T15:59:19.280372+0000', 'last_in_change': '2024-08-30T15:58:25.430425+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T15:59:08.355540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-08-30T15:59:21.696796+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6812', 'nonce': 1957399494}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6813', 'nonce': 1957399494}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 1957399494}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6814', 'nonce': 1957399494}]}, 'public_addr': '172.21.15.50:6812/1957399494', 'cluster_addr': '172.21.15.50:6813/1957399494', 'heartbeat_back_addr': '172.21.15.50:6815/1957399494', 'heartbeat_front_addr': '172.21.15.50:6814/1957399494', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0fe11a1c-4326-4bfd-a2a4-bff14023f301', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6808', 'nonce': 3208178756}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6809', 'nonce': 3208178756}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 3208178756}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6810', 'nonce': 3208178756}]}, 'public_addr': '172.21.15.50:6808/3208178756', 'cluster_addr': '172.21.15.50:6809/3208178756', 'heartbeat_back_addr': '172.21.15.50:6811/3208178756', 'heartbeat_front_addr': '172.21.15.50:6810/3208178756', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a9d5354b-3600-41ed-80c5-a3439da7aca6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6804', 'nonce': 287018800}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6805', 'nonce': 287018800}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 287018800}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6806', 'nonce': 287018800}]}, 'public_addr': '172.21.15.50:6804/287018800', 'cluster_addr': '172.21.15.50:6805/287018800', 'heartbeat_back_addr': '172.21.15.50:6807/287018800', 'heartbeat_front_addr': '172.21.15.50:6806/287018800', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '78bcc398-a7e4-4efc-9836-1dfb94407c9c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6800', 'nonce': 1140717714}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6801', 'nonce': 1140717714}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 1140717714}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6802', 'nonce': 1140717714}]}, 'public_addr': '172.21.15.50:6800/1140717714', 'cluster_addr': '172.21.15.50:6801/1140717714', 'heartbeat_back_addr': '172.21.15.50:6803/1140717714', 'heartbeat_front_addr': '172.21.15.50:6802/1140717714', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '53a1060b-7985-4c2c-820e-64b158056113', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 1468073345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6806', 'nonce': 1468073345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6808', 'nonce': 1468073345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 1468073345}]}, 'public_addr': '172.21.15.94:6805/1468073345', 'cluster_addr': '172.21.15.94:6806/1468073345', 'heartbeat_back_addr': '172.21.15.94:6808/1468073345', 'heartbeat_front_addr': '172.21.15.94:6807/1468073345', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '853a03f7-227d-4f98-9e5c-c29896820746', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 3876025616}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6810', 'nonce': 3876025616}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6812', 'nonce': 3876025616}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 3876025616}]}, 'public_addr': '172.21.15.94:6809/3876025616', 'cluster_addr': '172.21.15.94:6810/3876025616', 'heartbeat_back_addr': '172.21.15.94:6812/3876025616', 'heartbeat_front_addr': '172.21.15.94:6811/3876025616', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bc978060-bf28-4027-8bba-7d6d8659bfc5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 128519122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6802', 'nonce': 128519122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6804', 'nonce': 128519122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 128519122}]}, 'public_addr': '172.21.15.94:6801/128519122', 'cluster_addr': '172.21.15.94:6802/128519122', 'heartbeat_back_addr': '172.21.15.94:6804/128519122', 'heartbeat_front_addr': '172.21.15.94:6803/128519122', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '45df5c6f-b124-46d1-9efb-62b78bc584ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 68721277}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6814', 'nonce': 68721277}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6816', 'nonce': 68721277}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 68721277}]}, 'public_addr': '172.21.15.94:6813/68721277', 'cluster_addr': '172.21.15.94:6814/68721277', 'heartbeat_back_addr': '172.21.15.94:6816/68721277', 'heartbeat_front_addr': '172.21.15.94:6815/68721277', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.911895+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:04.786543+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:04.197952+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.680847+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:01.937600+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:02.082761+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.401025+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.129371+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T16:00:09.648 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:00:09.649 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:09.649 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:09.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:09.687 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:09.685+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:09.788 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:09.786+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:09.818 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:09.818 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:09.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:09.912 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:09.911+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:09.991 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:09.989+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:10.023 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:10.023 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:10.032 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:10.090 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:10.088+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:10.160 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:10.160 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:10.169 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:10.212 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:10.209+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:10.292 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:10.290+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:10.294 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:00:10.295 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:00:10.308 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-08-30T16:00:09.877926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218736,"kb_used_data":2532,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755984,"statfs":{"total":773094113280,"available":772870127616,"internally_reserved":0,"allocated":2592768,"data_stored":1779212,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":65,"apply_latency_ms":65,"commit_latency_ns":65000000,"apply_latency_ns":65000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.518933"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-08-30T16:00:07.114690+0000","last_change":"2024-08-30T15:59:27.646756+0000","last_active":"2024-08-30T16:00:07.114690+0000","last_peered":"2024-08-30T16:00:07.114690+0000","last_clean":"2024-08-30T16:00:07.114690+0000","last_became_active":"2024-08-30T15:59:25.342321+0000","last_became_peered":"2024-08-30T15:59:25.342321+0000","last_unstale":"2024-08-30T16:00:07.114690+0000","last_undegraded":"2024-08-30T16:00:07.114690+0000","last_fullsized":"2024-08-30T16:00:07.114690+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:53:42.927368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.116914037,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-08-30T16:00:05.446708+0000","last_change":"2024-08-30T15:59:28.156737+0000","last_active":"2024-08-30T16:00:05.446708+0000","last_peered":"2024-08-30T16:00:05.446708+0000","last_clean":"2024-08-30T16:00:05.446708+0000","last_became_active":"2024-08-30T15:59:25.662524+0000","last_became_peered":"2024-08-30T15:59:25.662524+0000","last_unstale":"2024-08-30T16:00:05.446708+0000","last_undegraded":"2024-08-30T16:00:05.446708+0000","last_fullsized":"2024-08-30T16:00:05.446708+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:05:30.092449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064329163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-08-30T16:00:07.110060+0000","last_change":"2024-08-30T15:59:27.633478+0000","last_active":"2024-08-30T16:00:07.110060+0000","last_peered":"2024-08-30T16:00:07.110060+0000","last_clean":"2024-08-30T16:00:07.110060+0000","last_became_active":"2024-08-30T15:59:24.572730+0000","last_became_peered":"2024-08-30T15:59:24.572730+0000","last_unstale":"2024-08-30T16:00:07.110060+0000","last_undegraded":"2024-08-30T16:00:07.110060+0000","last_fullsized":"2024-08-30T16:00:07.110060+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:36:20.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-08-30T16:00:06.604883+0000","last_change":"2024-08-30T15:59:27.584250+0000","last_active":"2024-08-30T16:00:06.604883+0000","last_peered":"2024-08-30T16:00:06.604883+0000","last_clean":"2024-08-30T16:00:06.604883+0000","last_became_active":"2024-08-30T15:59:24.687233+0000","last_became_peered":"2024-08-30T15:59:24.687233+0000","last_unstale":"2024-08-30T16:00:06.604883+0000","last_undegraded":"2024-08-30T16:00:06.604883+0000","last_fullsized":"2024-08-30T16:00:06.604883+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:49:09.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":29,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-08-30T16:00:07.111965+0000","last_change":"2024-08-30T15:59:27.623039+0000","last_active":"2024-08-30T16:00:07.111965+0000","last_peered":"2024-08-30T16:00:07.111965+0000","last_clean":"2024-08-30T16:00:07.111965+0000","last_became_active":"2024-08-30T15:59:24.688486+0000","last_became_peered":"2024-08-30T15:59:24.688486+0000","last_unstale":"2024-08-30T16:00:07.111965+0000","last_undegraded":"2024-08-30T16:00:07.111965+0000","last_fullsized":"2024-08-30T16:00:07.111965+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:22:51.391036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-08-30T16:00:07.101541+0000","last_change":"2024-08-30T15:59:27.825532+0000","last_active":"2024-08-30T16:00:07.101541+0000","last_peered":"2024-08-30T16:00:07.101541+0000","last_clean":"2024-08-30T16:00:07.101541+0000","last_became_active":"2024-08-30T15:59:25.469390+0000","last_became_peered":"2024-08-30T15:59:25.469390+0000","last_unstale":"2024-08-30T16:00:07.101541+0000","last_undegraded":"2024-08-30T16:00:07.101541+0000","last_fullsized":"2024-08-30T16:00:07.101541+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T17:46:39.688911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-08-30T16:00:07.105165+0000","last_change":"2024-08-30T15:59:27.583986+0000","last_active":"2024-08-30T16:00:07.105165+0000","last_peered":"2024-08-30T16:00:07.105165+0000","last_clean":"2024-08-30T16:00:07.105165+0000","last_became_active":"2024-08-30T15:59:24.528536+0000","last_became_peered":"2024-08-30T15:59:24.528536+0000","last_unstale":"2024-08-30T16:00:07.105165+0000","last_undegraded":"2024-08-30T16:00:07.105165+0000","last_fullsized":"2024-08-30T16:00:07.105165+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:18:40.602710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-08-30T16:00:07.108091+0000","last_change":"2024-08-30T15:59:27.636613+0000","last_active":"2024-08-30T16:00:07.108091+0000","last_peered":"2024-08-30T16:00:07.108091+0000","last_clean":"2024-08-30T16:00:07.108091+0000","last_became_active":"2024-08-30T15:59:25.021268+0000","last_became_peered":"2024-08-30T15:59:25.021268+0000","last_unstale":"2024-08-30T16:00:07.108091+0000","last_undegraded":"2024-08-30T16:00:07.108091+0000","last_fullsized":"2024-08-30T16:00:07.108091+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T00:56:28.767086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113256018,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":26,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-08-30T16:00:01.883967+0000","last_change":"2024-08-30T15:59:21.623783+0000","last_active":"2024-08-30T16:00:01.883967+0000","last_peered":"2024-08-30T16:00:01.883967+0000","last_clean":"2024-08-30T16:00:01.883967+0000","last_became_active":"2024-08-30T15:59:20.848924+0000","last_became_peered":"2024-08-30T15:59:20.848924+0000","last_unstale":"2024-08-30T16:00:01.883967+0000","last_undegraded":"2024-08-30T16:00:01.883967+0000","last_fullsized":"2024-08-30T16:00:01.883967+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:14:52.575421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":795680,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280530,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":508420,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8320000000000007}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.962999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.164999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.550000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.431000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.946}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.786999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313234,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":323584,"data_stored":127772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.566000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.443}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1769999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8420000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.1479999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.702999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.3230000000000004}]}]},{"osd":6,"up_from":21,"seq":90194313233,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":327680,"data_stored":127791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1890000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.875}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7349999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.641999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.487}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.460000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.1}]}]},{"osd":3,"up_from":21,"seq":90194313233,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":327680,"data_stored":127791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.6370000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8380000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.917}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.690999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.207000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.606000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.945}]}]},{"osd":4,"up_from":20,"seq":85899345937,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":119176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1289999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.818}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.459}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.161999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.032}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.675000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1839999999999993}]}]},{"osd":2,"up_from":16,"seq":68719476751,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":315392,"data_stored":123474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9480000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.654}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.34}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.364000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.937999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.004999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509454,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":119176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.352}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.045000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":36.715000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":56.200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.265000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":34.569000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":50.229999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":360448,"data_stored":525612,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:00:07 2024","interfaces":[{"interface":"back","average":{"1min":14.65,"5min":14.65,"15min":14.65},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.5279999999999996},"max":{"1min":111.313,"5min":111.313,"15min":111.313},"last":9.4440000000000008},{"interface":"front","average":{"1min":16.170999999999999,"5min":16.170999999999999,"15min":16.170999999999999},"min":{"1min":4.891,"5min":4.891,"15min":4.891},"max":{"1min":107.937,"5min":107.937,"15min":107.937},"last":10.012}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":37.576999999999998}]},{"osd":2,"last update":"Fri Aug 30 16:00:07 2024","interfaces":[{"interface":"back","average":{"1min":19.32,"5min":19.32,"15min":19.32},"min":{"1min":4.7220000000000004,"5min":4.7220000000000004,"15min":4.7220000000000004},"max":{"1min":84.317999999999998,"5min":84.317999999999998,"15min":84.317999999999998},"last":17.727},{"interface":"front","average":{"1min":17.359999999999999,"5min":17.359999999999999,"15min":17.359999999999999},"min":{"1min":6.0279999999999996,"5min":6.0279999999999996,"15min":6.0279999999999996},"max":{"1min":99.884,"5min":99.884,"15min":99.884},"last":22.635000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.422999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.338000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.245999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":34.021000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:00:10.309 INFO:tasks.thrashosds.thrasher:Setting 2.3 to [7, 6] 2024-08-30T16:00:10.310 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.3', '7', '6'] 2024-08-30T16:00:10.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.3 7 6 2024-08-30T16:00:10.339 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:10.340 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:10.350 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:00:10.397 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:10.395+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:10.417 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-08-30T16:00:10.483 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:10.483 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:10.493 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:00:10.494 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:10.492+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:10.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:00:10.595 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:10.593+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:10.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:00:10.666 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:10.666 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:10.675 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:00:10.700 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:10.699+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:10.797 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:10.796+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:10.841 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:10.841 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:10.851 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:00:10.898 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:10.897+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:10.977 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:10.977 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:10.986 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:00:10.999 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:10.998+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:11.100 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:11.099+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:11.112 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:11.112 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:11.121 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:00:11.201 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:11.200+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:11.246 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:11.246 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:11.255 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:00:11.302 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:11.300+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:11.392 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:11.392 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:11.402 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:00:11.403 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:11.402+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:11.504 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:11.503+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:11.587 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:11.587 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:11.596 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:00:11.637 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:11.636+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:11.707 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:11.705+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:11.765 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:11.765 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:11.775 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:00:11.775 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:00:11.776 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:00:11.807 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:11.806+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:11.867 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:00:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:00:11.928 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:11.926+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:11.941 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:11.941 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:11.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:00:11.953 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:00:11.971 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-08-30T16:00:11.971 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-30T16:00:11.971 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:00:11.985 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:00:11.986 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:00:12.009 INFO:tasks.ceph.mon.c.smithi050.stderr:2024-08-30T16:00:11.988+0000 a084640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 22533) UID: 0 2024-08-30T16:00:12.010 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:12.008+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:12.010 INFO:tasks.ceph.mon.c.smithi050.stderr:2024-08-30T16:00:11.991+0000 a084640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-08-30T16:00:12.110 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:12.109+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:12.117 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:12.117 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:12.117 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:12.117 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:12.129 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:00:12.129 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:00:12.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:00:12.225 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:12.223+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:12.225 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:00:12.226 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:00:12.316 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:12.316 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:12.316 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:12.316 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:12.317 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:12.317 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:12.315+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:12.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:00:12.328 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:00:12.328 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:00:12.414 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:00:12.414 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:00:12.414 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:12.413+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:12.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-08-30T16:00:12.499 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:12.500 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:12.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:12.500 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:00:12.500 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:12.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:12.511 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:00:12.512 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:00:12.533 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:12.529+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:12.616 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:12.614+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:12.647 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:12.648 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:12.648 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:12.648 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:12.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:12.717 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:12.716+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:12.818 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:12.816+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:12.830 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:12.830 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:12.830 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:12.830 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:12.830 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:12.840 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:12.919 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:12.917+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:12.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-08-30T16:00:12.974 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.699502889999998, 2024-08-30T16:00:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:00:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:00:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:00:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:00:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:00:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:00:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:00:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:00:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:00:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:00:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:00:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:00:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:00:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:00:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:00:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.426575528999997, 2024-08-30T16:00:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:00:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:00:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:00:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:00:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:00:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:00:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:00:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:00:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:00:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:00:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:00:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:00:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:00:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:00:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:00:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:12.983 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:12.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:13.020 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:13.019+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:13.111 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:13.112 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:13.112 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:13.112 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:13.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:13.121 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:13.119+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:13.222 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:13.220+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:13.248 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:13.248 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:13.248 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:13.248 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:13.248 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:13.257 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:13.323 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:13.322+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:13.386 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:13.386 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:13.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:13.387 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:13.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.387 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:00:13.387 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:13.387 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 48.009269476, 2024-08-30T16:00:13.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:00:13.387 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:13.387 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:13.387 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:13.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:13.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:13.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:13.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:13.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:13.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:00:13.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:00:13.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:13.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:00:13.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:00:13.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:13.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:00:13.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:00:13.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:13.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:00:13.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:00:13.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:13.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:00:13.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:00:13.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:00:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:00:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:00:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:00:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:00:13.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:00:13.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.392 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:13.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.392 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:13.392 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:13.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:13.437 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:13.436+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:13.525 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:13.524+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:13.569 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:13.569 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:13.569 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:13.569 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:13.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:13.626 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:13.625+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:13.727 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:13.726+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:13.745 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:13.745 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:13.745 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:13.745 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:13.745 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:13.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:13.828 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:13.827+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:13.903 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 49.454173920000002, 2024-08-30T16:00:13.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:00:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:00:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:00:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:13.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:00:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:00:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:00:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:00:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:00:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:00:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:00:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:00:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:00:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:00:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:00:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:00:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:00:13.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:00:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:00:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:00:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:00:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:00:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:13.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:00:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:00:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:00:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:00:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:00:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:00:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:00:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:00:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:00:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 48.338307585000003, 2024-08-30T16:00:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:00:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:13.910 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:13.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:00:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:00:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:00:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:00:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:00:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:00:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:00:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:00:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:00:13.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:00:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:00:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:00:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:00:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 48.278716086999999, 2024-08-30T16:00:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:00:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:00:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:13.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:00:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:00:13.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:00:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:00:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:00:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:00:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:13.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:00:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:00:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:00:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:00:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:00:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:00:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.918 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:00:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 48.2520174, 2024-08-30T16:00:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:00:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:13.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:00:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:00:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:00:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:00:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:00:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:13.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:00:13.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:00:13.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:13.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:00:13.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:00:13.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:13.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:00:13.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:00:13.923 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.923 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:13.923 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.923 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:13.923 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:13.923 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.384400872000001, 2024-08-30T16:00:13.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:00:13.923 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:13.923 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:13.923 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:00:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:13.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:00:13.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:13.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:13.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:00:13.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:00:13.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:13.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:00:13.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:00:13.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:13.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:00:13.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:00:13.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:13.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:00:13.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:00:13.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:13.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:00:13.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:00:13.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:13.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:00:13.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:00:13.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:13.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:00:13.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:00:13.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:13.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:00:13.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:00:13.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:13.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:00:13.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:00:13.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:13.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:00:13.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:00:13.929 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.929 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:13.929 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.929 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:00:13.929 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:13.929 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.225153792999997, 2024-08-30T16:00:13.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:00:13.929 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:13.929 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:13.929 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:13.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:13.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:13.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:13.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:13.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:13.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:00:13.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:00:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:00:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:00:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:00:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:00:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:13.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:00:13.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:00:13.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:13.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:00:13.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:00:13.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:13.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:00:13.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:00:13.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:00:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:00:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.948979727000001, 2024-08-30T16:00:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:00:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:13.933 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:13.934 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:13.934 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:13.934 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:13.934 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:13.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.934 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:13.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:13.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:13.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:13.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:13.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:13.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:00:13.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:13.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:13.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:00:13.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:00:13.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:13.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:00:13.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:00:13.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:13.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:00:13.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:00:13.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:13.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:00:13.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:00:13.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:13.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:00:13.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:00:13.937 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.937 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:13.937 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.937 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:13.937 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:13.937 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.290688148000001, 2024-08-30T16:00:13.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:00:13.938 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:13.938 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:13.938 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:13.938 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:13.938 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:13.938 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:00:13.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.938 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:13.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:13.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:13.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:13.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:13.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:13.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:00:13.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:13.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:13.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:00:13.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:00:13.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:13.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:00:13.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:00:13.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:13.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:00:13.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:00:13.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:13.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:00:13.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:00:13.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:13.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:00:13.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:00:13.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:13.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:00:13.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:00:13.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:13.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:00:13.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:00:13.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:13.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:00:13.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:00:13.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:13.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:00:13.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:00:13.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:13.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:00:13.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:00:13.943 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.943 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:13.943 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.943 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:00:13.944 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:13.944 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.194774864000003, 2024-08-30T16:00:13.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:00:13.944 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:13.944 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:13.944 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:13.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:13.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:13.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:00:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:00:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:00:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:13.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:00:13.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:00:13.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:13.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:00:13.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:00:13.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:13.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:00:13.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:00:13.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:13.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:00:13.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:00:13.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:13.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:13.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:13.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:00:13.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:00:13.947 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.947 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:13.947 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.947 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:13.948 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:13.948 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:13.949 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:13.950 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:13.948+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:14.051 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:14.049+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:14.074 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:14.074 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:14.074 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:14.074 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:14.084 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:14.151 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:14.150+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:14.250 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:14.250 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:14.250 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:14.252 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:14.250+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:14.259 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:00:14.353 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:14.351+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:14.424 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:14.424 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:14.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:14.424 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:00:14.424 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:14.432 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:00:14.474 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:14.472+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:14.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:00:14.555 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:14.553+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:14.596 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:14.596 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:14.596 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:14.596 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:14.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:00:14.690 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:14.688+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:14.757 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:14.755+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:14.769 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:14.769 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:14.769 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:14.779 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:00:14.863 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:14.861+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:14.959 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:14.958+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:14.990 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:14.990 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:14.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:14.990 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:14.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.990 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:14.990 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:14.990 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 64.128531081999995, 2024-08-30T16:00:14.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:00:14.990 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:14.991 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:14.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:14.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:14.991 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:14.991 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:00:14.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.991 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:14.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:14.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:14.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:14.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:14.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:14.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:14.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:14.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:00:14.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:00:14.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:14.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:00:14.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:00:14.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:14.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:00:14.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:00:14.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:14.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:00:14.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:00:14.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:14.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:00:14.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:00:14.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:00:14.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:00:14.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:00:14.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:14.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:00:14.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:00:14.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:14.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:00:14.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:00:14.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:14.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:00:14.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:00:14.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:14.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:00:14.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:00:14.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:14.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:00:14.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:00:14.997 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:14.997 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:14.997 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:14.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.997 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:14.997 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:14.997 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.212115062000002, 2024-08-30T16:00:14.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:00:14.998 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:14.998 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:14.998 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:14.998 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:14.998 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:14.998 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:00:14.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.998 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:14.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:14.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:14.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:14.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:14.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:14.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:14.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:14.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:00:14.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:14.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:14.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:14.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:00:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:00:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:00:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:00:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:00:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:00:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:00:15.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:00:15.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:00:15.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:00:15.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:00:15.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:00:15.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:00:15.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:00:15.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:00:15.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:00:15.002 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.002 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.003 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.003 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:15.003 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:15.003 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 63.210410508999999, 2024-08-30T16:00:15.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:00:15.003 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.003 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.003 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.003 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:00:15.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.004 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:15.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:15.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:00:15.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:00:15.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:00:15.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:00:15.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:00:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:00:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:00:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:00:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:00:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:00:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:00:15.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:00:15.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:00:15.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:00:15.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:00:15.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:00:15.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:00:15.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:00:15.008 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.008 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.008 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.008 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.008 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:15.009 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.790956702000003, 2024-08-30T16:00:15.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:00:15.009 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.009 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.009 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.009 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.009 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.009 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:00:15.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.009 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:15.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:15.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:00:15.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:00:15.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:00:15.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:00:15.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:00:15.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:00:15.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:00:15.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:00:15.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:00:15.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:00:15.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:00:15.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:00:15.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:00:15.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:00:15.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:00:15.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:00:15.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:00:15.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:00:15.013 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.013 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.013 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.014 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.014 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:15.014 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.774068528000001, 2024-08-30T16:00:15.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:00:15.014 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.014 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.014 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:00:15.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.014 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:15.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:15.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:00:15.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:00:15.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:00:15.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:00:15.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:00:15.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:00:15.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:00:15.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:00:15.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:00:15.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:00:15.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:00:15.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:00:15.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:00:15.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:00:15.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:00:15.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:00:15.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:00:15.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:00:15.019 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.019 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.019 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.020 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.020 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:15.020 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.761068459000001, 2024-08-30T16:00:15.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:00:15.020 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.020 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.020 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:00:15.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.020 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:15.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:15.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:00:15.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:00:15.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:00:15.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:00:15.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:00:15.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:00:15.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:00:15.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:00:15.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:00:15.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:00:15.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:00:15.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:00:15.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:00:15.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:00:15.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:00:15.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:00:15.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:00:15.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:00:15.025 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.025 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.025 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.025 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.025 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:15.025 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.758576167000001, 2024-08-30T16:00:15.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:00:15.025 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.025 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.026 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.026 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.026 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.026 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:00:15.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.026 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:15.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:15.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:00:15.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:15.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:00:15.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:00:15.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:00:15.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:00:15.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:00:15.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:00:15.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:00:15.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:00:15.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:00:15.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:00:15.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:00:15.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:00:15.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:00:15.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:00:15.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:00:15.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:00:15.030 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.031 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.031 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.031 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.031 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:15.031 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.757818718999999, 2024-08-30T16:00:15.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:00:15.031 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.031 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.031 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.031 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.031 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.032 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:00:15.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.032 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:15.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:15.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:00:15.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:00:15.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:00:15.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:00:15.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:00:15.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:00:15.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:00:15.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:00:15.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:00:15.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:00:15.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:00:15.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:00:15.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:00:15.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:00:15.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:00:15.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:00:15.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:00:15.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:00:15.036 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.036 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.036 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.036 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.036 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:15.036 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.757086268999998, 2024-08-30T16:00:15.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:00:15.037 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.037 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.037 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.037 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.037 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.037 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:00:15.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.037 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:15.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:15.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:00:15.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:15.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:00:15.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:00:15.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:00:15.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:00:15.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:00:15.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:00:15.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:00:15.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:00:15.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:00:15.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:00:15.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:00:15.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:00:15.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:00:15.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:00:15.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:00:15.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:00:15.041 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.042 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.042 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.042 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.042 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:15.042 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.754547469999999, 2024-08-30T16:00:15.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:00:15.042 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.042 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.043 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:00:15.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.043 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:15.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:15.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:00:15.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:15.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:00:15.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:00:15.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:00:15.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:00:15.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:00:15.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:00:15.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:00:15.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:00:15.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:00:15.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:00:15.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:00:15.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:00:15.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:00:15.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:00:15.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:00:15.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:00:15.047 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.047 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.047 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.047 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.047 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:15.047 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 62.753799469, 2024-08-30T16:00:15.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:00:15.048 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.048 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.048 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.048 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.048 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.048 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:00:15.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.048 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:15.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:00:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:00:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:00:15.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:00:15.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:00:15.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:00:15.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:00:15.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:00:15.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:00:15.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:00:15.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:00:15.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:00:15.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:00:15.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:00:15.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:00:15.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:00:15.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:00:15.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:00:15.053 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.053 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.053 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.053 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.053 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:15.053 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.981682997, 2024-08-30T16:00:15.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:00:15.053 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.053 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.053 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.054 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.054 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.054 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:00:15.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.054 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:15.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:15.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:00:15.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:00:15.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:00:15.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:00:15.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:00:15.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:00:15.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:00:15.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:00:15.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:00:15.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:00:15.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:00:15.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:00:15.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:00:15.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:00:15.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:00:15.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:00:15.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:00:15.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:00:15.058 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.058 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.058 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.058 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.059 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:15.059 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.979625851000002, 2024-08-30T16:00:15.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:00:15.059 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.059 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.059 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.059 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.059 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.059 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:00:15.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.059 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:15.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:15.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:00:15.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:15.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:00:15.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:00:15.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:00:15.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:00:15.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:00:15.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:00:15.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:00:15.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:00:15.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:00:15.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:00:15.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:00:15.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:00:15.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:00:15.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:00:15.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:00:15.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:00:15.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.064 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.064 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.064 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:15.064 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.716681350999998, 2024-08-30T16:00:15.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:00:15.065 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.065 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.065 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.065 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.065 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.065 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:00:15.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.065 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:15.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:00:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:00:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:00:15.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:00:15.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:00:15.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:00:15.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:00:15.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:00:15.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:00:15.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:00:15.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:00:15.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:00:15.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:00:15.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:00:15.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:00:15.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:00:15.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:00:15.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:00:15.069 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.069 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.069 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.070 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.070 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:15.070 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.715589848, 2024-08-30T16:00:15.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:00:15.070 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.070 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.070 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.070 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.070 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.070 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:00:15.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.071 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:15.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:15.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:00:15.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:15.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:00:15.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:00:15.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:00:15.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:00:15.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:00:15.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:00:15.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:00:15.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:00:15.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:00:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:00:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:00:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:00:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:00:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:00:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:00:15.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:00:15.075 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.075 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.075 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.075 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.075 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:15.075 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.714840262999999, 2024-08-30T16:00:15.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:00:15.075 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.076 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.076 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.076 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.076 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.076 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:00:15.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.076 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:15.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:15.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:00:15.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:15.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:00:15.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:00:15.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:00:15.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:00:15.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:00:15.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:00:15.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:00:15.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:00:15.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:00:15.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:00:15.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:00:15.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:00:15.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:00:15.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:00:15.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:00:15.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:00:15.080 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.080 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.080 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.081 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.081 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:15.081 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.681108731000002, 2024-08-30T16:00:15.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:00:15.081 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.081 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.081 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.081 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.081 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.081 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:00:15.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.081 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:15.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:15.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:00:15.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:00:15.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:00:15.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:00:15.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:00:15.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:00:15.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:00:15.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:00:15.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:00:15.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:00:15.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:00:15.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:00:15.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:00:15.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:00:15.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:00:15.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:00:15.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:00:15.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:00:15.086 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.086 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.086 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.086 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.086 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:15.086 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.680582723000001, 2024-08-30T16:00:15.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:00:15.086 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.086 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.087 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.087 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.087 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.087 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:00:15.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.087 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:15.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:15.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:00:15.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:15.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:00:15.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:00:15.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:00:15.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:00:15.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:00:15.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:00:15.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:00:15.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:00:15.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:00:15.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:00:15.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:00:15.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:00:15.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:00:15.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:00:15.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:00:15.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:00:15.091 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.091 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.091 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.092 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:15.092 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:15.092 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.680127159999998, 2024-08-30T16:00:15.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:00:15.092 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.092 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.092 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.092 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.092 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.092 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:00:15.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.092 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:15.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:15.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:00:15.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:00:15.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:00:15.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:00:15.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:00:15.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:00:15.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:00:15.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:00:15.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:00:15.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:00:15.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:00:15.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:00:15.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:00:15.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:00:15.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:00:15.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:00:15.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:00:15.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:00:15.097 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.097 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.097 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.097 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:00:15.097 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:15.097 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.009611843999998, 2024-08-30T16:00:15.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:00:15.097 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.098 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.098 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:15.098 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:15.098 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:15.098 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:00:15.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.098 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:15.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:15.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:00:15.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:15.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:00:15.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:00:15.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:00:15.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:00:15.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:00:15.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:00:15.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:00:15.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:00:15.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:00:15.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:00:15.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:15.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:00:15.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:00:15.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:00:15.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:00:15.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:00:15.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:00:15.102 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.102 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.102 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.102 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.102 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.102 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:15.104 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:00:15.104 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:15.103+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:15.205 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:15.203+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:15.269 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:15.270 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:15.278 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:00:15.321 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:15.319+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:15.407 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:15.406+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:15.444 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:15.445 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:15.445 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:15.445 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:15.445 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:15.453 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:00:15.538 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:15.536+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:15.610 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:15.609+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:15.621 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:15.621 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:15.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:15.622 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:15.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.622 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:00:15.622 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:15.622 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 49.091498561000002, 2024-08-30T16:00:15.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:00:15.622 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.622 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.622 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:15.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:15.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:00:15.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:00:15.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:00:15.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:00:15.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:00:15.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:00:15.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:00:15.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:00:15.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:00:15.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:00:15.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:00:15.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:00:15.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:00:15.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:00:15.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:00:15.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:00:15.626 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.626 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.626 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.626 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:00:15.626 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:15.627 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 47.973050415000003, 2024-08-30T16:00:15.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:00:15.627 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:15.627 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:15.627 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:15.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:15.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:15.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:00:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:00:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:00:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:00:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:00:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:00:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:00:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:00:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:00:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:00:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:00:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:00:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:00:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:00:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:00:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:00:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:15.632 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:15.633 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:00:15.717 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:15.715+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:15.796 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:15.796 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:15.796 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:15.796 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:15.805 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:00:15.847 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:15.845+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:15.912 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:15.911+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:15.967 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:15.968 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:15.968 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:15.968 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:15.968 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:15.976 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:00:16.021 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:16.019+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:16.102 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:16.102 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:16.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:16.102 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:16.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:16.103 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:16.103 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:16.103 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 47.724048132999997, 2024-08-30T16:00:16.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:00:16.103 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:16.103 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:00:16.103 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:16.103 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:00:16.103 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:16.103 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:00:16.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:16.103 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:16.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:16.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:16.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:16.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:16.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:16.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:16.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:16.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:16.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:16.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:16.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:16.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:16.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:00:16.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:00:16.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:16.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:16.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:16.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:00:16.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:00:16.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:16.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:16.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:16.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:00:16.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:00:16.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:16.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:16.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:16.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:00:16.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:00:16.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:16.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:16.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:16.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:00:16.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:00:16.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:16.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:16.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:16.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:00:16.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:00:16.107 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:16.107 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:16.107 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:16.107 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:16.107 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:16.107 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:16.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:00:16.153 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:16.151+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:16.214 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:16.212+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:16.252 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:16.252 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:16.252 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:16.252 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:16.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:00:16.315 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:16.314+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:16.416 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:16.415+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:16.442 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:16.442 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:16.442 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:16.442 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:16.442 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:16.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:00:16.517 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:16.516+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:16.579 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:16.579 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:16.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:16.579 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:00:16.579 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:16.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:16.618 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:16.617+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:16.715 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:16.715 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:16.715 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:16.715 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:16.720 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:16.718+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:16.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:16.821 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:16.819+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:16.852 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:16.852 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:16.852 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:16.852 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:16.853 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:16.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:16.945 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:16.944+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:17.022 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:17.021+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:17.033 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:17.033 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:17.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:17.033 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:17.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.034 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:17.034 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:17.034 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.759169923000002, 2024-08-30T16:00:17.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:00:17.034 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:17.034 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:17.034 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:17.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:17.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:17.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:17.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:17.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:17.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:17.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:17.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:00:17.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:00:17.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:17.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:00:17.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:00:17.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:17.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:00:17.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:00:17.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:17.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:00:17.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:00:17.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:17.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:00:17.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:00:17.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:17.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:00:17.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:00:17.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:17.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:00:17.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:00:17.038 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:17.038 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:17.038 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:17.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.038 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:17.038 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:17.038 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.486242562000001, 2024-08-30T16:00:17.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:00:17.038 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:17.038 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:17.038 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:17.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:17.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:17.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:17.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:17.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:17.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:17.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:17.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:00:17.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:00:17.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:17.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:00:17.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:00:17.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:17.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:00:17.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:00:17.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:17.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:00:17.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:00:17.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:17.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:00:17.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:00:17.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:17.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:00:17.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:00:17.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:17.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:00:17.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:00:17.042 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:17.042 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:17.042 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:17.042 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:17.042 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:17.042 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:17.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:17.127 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:17.126+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:17.212 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:17.212 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:17.213 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:17.213 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:17.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:17.224 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:17.223+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:17.325 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:17.323+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:17.389 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:17.389 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:17.389 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:17.389 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:17.389 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:17.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:17.443 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:17.440+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:17.527 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:17.524+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:17.527 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:00:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 52.149889741000003, 2024-08-30T16:00:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:00:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:00:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:00:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:00:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:00:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:00:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:00:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:00:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:00:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:00:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:00:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:00:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:00:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:00:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:00:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:00:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:00:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:17.533 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:17.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:17.627 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:17.626+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:17.712 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:17.712 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:17.712 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:17.712 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:17.722 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:17.728 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:17.726+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:17.829 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:17.827+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:17.849 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:17.849 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:17.849 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:17.850 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:17.850 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:17.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:17.930 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:17.929+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:18.031 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:18.030+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:18.046 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 53.598166687999999, 2024-08-30T16:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:00:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:00:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 52.482300352999999, 2024-08-30T16:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:00:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 52.422708855000003, 2024-08-30T16:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:00:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 52.396010167999997, 2024-08-30T16:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:00:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.528393639999997, 2024-08-30T16:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:00:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:00:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:00:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:00:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:00:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:00:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:00:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:00:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:00:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:00:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:00:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:00:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:00:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:00:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:00:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:00:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:00:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.369146561000001, 2024-08-30T16:00:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:00:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:00:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:00:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:00:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:00:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:00:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:00:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:00:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:00:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:00:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:00:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:00:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:00:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:00:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:00:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.092972494999998, 2024-08-30T16:00:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:00:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:00:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:00:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:00:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:00:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:00:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:00:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:00:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:00:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:00:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:00:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:00:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 50.434680915999998, 2024-08-30T16:00:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:00:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:18.084 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:18.084 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:18.084 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:00:18.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.084 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:18.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:18.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:18.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:18.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:18.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:18.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:00:18.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:18.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:18.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:00:18.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:00:18.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:18.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:00:18.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:00:18.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:18.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:00:18.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:00:18.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:18.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:00:18.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:00:18.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:18.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:00:18.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:00:18.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:18.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:00:18.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:00:18.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:18.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:00:18.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:00:18.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:18.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:00:18.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:00:18.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:18.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:00:18.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:00:18.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:18.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:00:18.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:00:18.089 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.089 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:18.089 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.089 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:00:18.089 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:18.089 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 50.338767632, 2024-08-30T16:00:18.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:00:18.090 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:18.090 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:18.090 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:18.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:18.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:18.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:18.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:18.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:18.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:18.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:00:18.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:18.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:18.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:00:18.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:00:18.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:18.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:00:18.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:00:18.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:18.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:00:18.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:00:18.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:18.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:00:18.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:00:18.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:18.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:00:18.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:00:18.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:18.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:18.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:18.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:00:18.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:00:18.093 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.093 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:18.093 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.093 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:18.094 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:18.094 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:18.094 INFO:tasks.ceph.mon.c:Stopped 2024-08-30T16:00:18.094 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2024-08-30T16:00:18.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:00:18.096 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:18.140 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:18.138+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:18.237 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:18.233+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:18.253 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:18.253 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:18.253 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:18.253 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:18.262 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:18.334 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:18.332+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:18.426 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:18.426 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:18.427 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:18.427 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:18.427 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:18.435 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:00:18.436 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:18.433+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:18.536 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:18.534+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:18.559 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:18.559 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:18.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:18.559 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:00:18.559 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:18.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:00:18.637 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:18.635+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:18.698 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:18.698 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:18.698 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:18.698 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:18.707 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:00:18.751 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:18.749+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:18.833 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:18.833 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:18.833 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:18.833 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:18.834 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:18.839 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:18.837+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:18.842 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:00:18.940 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:18.938+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:19.041 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:19.039+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:19.053 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:19.053 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:19.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:19.053 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:19.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.054 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:19.054 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:19.054 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 68.192235874000005, 2024-08-30T16:00:19.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:00:19.054 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.054 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.054 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.054 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.054 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.054 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:00:19.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.054 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:19.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:19.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:00:19.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:00:19.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:00:19.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:00:19.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:00:19.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:00:19.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:00:19.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:00:19.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:00:19.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:00:19.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:00:19.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:00:19.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:00:19.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:00:19.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:00:19.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:00:19.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:00:19.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:00:19.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:00:19.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:00:19.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:00:19.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:00:19.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:00:19.060 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.060 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.060 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.060 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:19.060 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:19.060 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 67.275819854000005, 2024-08-30T16:00:19.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:00:19.061 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.061 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.061 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.061 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.061 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.061 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:00:19.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.061 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:19.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:19.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:00:19.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:19.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:00:19.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:00:19.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:00:19.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:00:19.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:00:19.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:00:19.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:00:19.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:00:19.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:00:19.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:00:19.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:00:19.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:00:19.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:00:19.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:00:19.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:00:19.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:00:19.065 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 67.274115300999995, 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:00:19.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.067 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:19.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:19.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:00:19.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:00:19.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:00:19.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:00:19.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:00:19.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:00:19.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:00:19.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:00:19.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:00:19.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:00:19.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:00:19.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:00:19.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:00:19.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:00:19.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.071 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:19.072 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.854661493999998, 2024-08-30T16:00:19.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:00:19.072 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.072 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.072 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:00:19.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.072 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:19.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:19.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:00:19.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:00:19.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:00:19.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:00:19.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:00:19.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:00:19.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:00:19.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:00:19.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:00:19.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:00:19.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:00:19.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:00:19.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.076 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.077 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.077 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.077 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:19.077 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.837773319999997, 2024-08-30T16:00:19.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:00:19.077 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.077 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.077 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.077 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.077 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.077 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:00:19.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:00:19.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:00:19.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:00:19.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:00:19.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:00:19.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:00:19.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:00:19.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:00:19.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:00:19.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:00:19.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:00:19.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:00:19.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:00:19.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:00:19.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:00:19.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:00:19.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:00:19.082 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.082 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.082 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.082 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.082 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:19.082 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.824773250999996, 2024-08-30T16:00:19.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:00:19.082 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.082 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.082 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.082 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.083 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.083 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:00:19.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.083 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:19.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:19.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:00:19.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:00:19.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:00:19.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:00:19.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:00:19.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:00:19.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:00:19.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:00:19.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:00:19.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:00:19.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:00:19.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:00:19.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:00:19.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:00:19.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:00:19.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:00:19.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:00:19.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:00:19.087 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.087 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.087 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.087 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.087 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:19.087 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.822280958999997, 2024-08-30T16:00:19.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:00:19.088 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.088 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.088 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.088 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.088 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.088 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:00:19.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.088 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:19.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:19.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:00:19.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:19.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:00:19.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:00:19.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:00:19.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:00:19.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:00:19.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:00:19.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:00:19.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:00:19.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:00:19.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:00:19.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:00:19.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:00:19.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:00:19.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:00:19.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:00:19.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:00:19.092 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.092 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.093 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.093 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.093 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:19.093 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.821523510999995, 2024-08-30T16:00:19.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:00:19.093 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.093 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.093 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.093 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.093 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.093 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:00:19.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.094 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:19.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:19.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:00:19.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:00:19.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:00:19.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:00:19.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:00:19.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:00:19.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:00:19.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:00:19.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:00:19.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:00:19.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:00:19.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:00:19.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:00:19.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:00:19.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:00:19.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:00:19.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:00:19.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:00:19.098 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.098 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.098 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.098 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.098 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:19.098 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.820791060999994, 2024-08-30T16:00:19.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:00:19.098 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.098 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.098 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.099 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.099 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.099 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:00:19.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.099 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:19.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:19.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:00:19.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:19.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:00:19.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:00:19.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:00:19.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:00:19.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:00:19.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:00:19.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:00:19.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:00:19.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:00:19.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:00:19.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:00:19.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:00:19.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:00:19.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:00:19.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:00:19.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:00:19.103 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.103 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.103 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.103 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.104 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:19.104 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.818252262000001, 2024-08-30T16:00:19.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:00:19.104 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.104 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.104 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.104 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.104 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.104 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:00:19.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.104 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:19.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:19.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:00:19.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:19.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:00:19.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:00:19.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:00:19.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:00:19.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:00:19.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:00:19.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:00:19.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:00:19.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:00:19.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:00:19.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:00:19.108 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.108 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.108 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.109 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.109 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:19.109 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.817504260999996, 2024-08-30T16:00:19.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:00:19.109 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.109 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.109 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.109 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.109 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.109 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:00:19.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.109 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:00:19.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:00:19.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:00:19.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:00:19.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:00:19.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:00:19.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:00:19.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:00:19.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:00:19.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:00:19.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:00:19.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:00:19.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:00:19.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:00:19.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:00:19.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:00:19.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:00:19.114 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.114 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.114 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.114 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.114 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:19.114 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.045387789000003, 2024-08-30T16:00:19.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:00:19.114 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.114 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:00:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:00:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:00:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:00:19.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:00:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:00:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:00:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:00:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:00:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:00:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:00:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:00:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:00:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:00:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:00:19.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:00:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:00:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:00:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:00:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 66.043330643000004, 2024-08-30T16:00:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:00:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:00:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:00:19.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:00:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:00:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:00:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:00:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:00:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:00:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:00:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:00:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:00:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:00:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:00:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:00:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:00:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:00:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:00:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:00:19.124 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.780386143000001, 2024-08-30T16:00:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:00:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.125 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:00:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:00:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:00:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:00:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:00:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:00:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:00:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:00:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:00:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:00:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:00:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:00:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:00:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:00:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:00:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:00:19.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:00:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:00:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:00:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.779294640000003, 2024-08-30T16:00:19.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:00:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:00:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:00:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:00:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:00:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:00:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:00:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:00:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:00:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:00:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:00:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:00:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:00:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:00:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:00:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:00:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:00:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:00:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:00:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.135 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.778545054999995, 2024-08-30T16:00:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:00:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.136 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:00:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:19.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:00:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:00:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:00:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:00:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:00:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:00:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:00:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:00:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:00:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:00:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:00:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.744813523000005, 2024-08-30T16:00:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:00:19.141 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:00:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:00:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:00:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:00:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:00:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:00:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:00:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:00:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:00:19.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:00:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:00:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:00:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:00:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:00:19.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.744287514999996, 2024-08-30T16:00:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:00:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:00:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:00:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:00:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:00:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:00:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:00:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:00:19.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:00:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:00:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:00:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:00:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:00:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:00:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:00:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:00:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:00:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:00:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:00:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.151 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.743831951999994, 2024-08-30T16:00:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:00:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.152 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:00:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:00:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:00:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:00:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:00:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:00:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:00:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:00:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:00:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:00:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:00:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:00:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:00:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:00:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:00:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:00:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:00:19.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:00:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:00:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:00:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.073316636000001, 2024-08-30T16:00:19.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:00:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:00:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:00:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:00:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:00:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:00:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:00:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:00:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:00:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:00:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:00:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:00:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:00:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:00:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:00:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:00:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:00:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:00:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:00:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.164 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:19.165 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:00:19.166 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:19.163+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:19.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-08-30T16:00:19.266 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:19.264+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:19.329 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:19.329 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:19.329 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:19.330 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:19.339 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:00:19.367 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:19.365+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:19.462 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:19.463 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:19.468 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:19.467+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:19.471 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:00:19.570 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:19.568+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:19.639 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:19.639 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:19.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:19.639 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:19.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.639 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:00:19.639 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:19.639 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 53.109618928000003, 2024-08-30T16:00:19.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:00:19.640 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.640 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.640 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:19.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:19.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:00:19.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:00:19.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:00:19.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:00:19.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:00:19.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:00:19.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:00:19.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:00:19.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:00:19.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:00:19.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:00:19.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:00:19.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:00:19.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:00:19.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:00:19.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:00:19.644 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.644 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.644 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.645 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:00:19.645 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:19.645 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 51.991170781999998, 2024-08-30T16:00:19.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:00:19.645 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:19.645 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:19.645 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:19.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:19.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:19.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:19.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:19.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:19.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:19.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:00:19.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:00:19.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:19.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:00:19.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:00:19.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:19.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:00:19.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:00:19.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:19.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:00:19.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:00:19.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:19.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:00:19.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:00:19.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:19.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:00:19.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:00:19.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:19.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:00:19.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:00:19.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:19.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:19.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:19.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:00:19.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:00:19.649 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.649 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.649 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.649 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:19.649 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:19.649 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:19.650 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:00:19.671 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:19.669+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:19.772 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:19.770+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:19.772 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:19.773 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:19.773 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:19.773 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:19.781 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:00:19.873 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:19.871+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:19.945 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:19.945 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:19.945 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:19.954 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:00:19.996 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:19.994+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:20.074 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:20.072+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:20.119 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:20.119 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:20.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:20.119 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:20.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:20.119 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:20.119 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:20.119 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 51.741192488999999, 2024-08-30T16:00:20.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:00:20.120 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:20.120 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:00:20.120 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:20.120 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:00:20.120 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:20.120 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:00:20.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:20.120 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:20.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:20.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:20.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:20.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:20.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:20.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:20.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:20.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:20.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:20.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:20.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:20.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:20.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:00:20.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:00:20.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:20.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:20.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:20.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:00:20.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:00:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:00:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:00:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:00:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:00:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:00:20.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:00:20.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:20.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:20.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:20.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:00:20.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:00:20.123 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:20.123 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:20.123 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:20.123 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:20.123 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:20.123 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:20.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:00:20.175 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:20.174+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:20.268 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:20.268 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:20.268 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:20.268 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:20.276 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:20.275+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:20.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:00:20.377 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:20.375+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:20.406 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:20.406 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:20.406 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:20.406 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:20.406 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:20.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:00:20.500 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:20.498+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:20.579 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:20.577+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:20.583 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:20.583 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:20.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:20.583 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:00:20.583 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:20.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:20.680 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:20.678+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:20.761 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:20.761 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:20.761 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:20.761 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:20.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:20.781 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:20.780+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:20.882 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:20.879+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:20.897 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:20.897 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:20.897 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:20.897 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:20.897 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:20.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:20.983 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:20.982+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:21.081 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:21.081 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:21.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:21.081 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:21.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.081 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:21.082 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:21.082 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 59.806755662999997, 2024-08-30T16:00:21.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:00:21.082 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:21.082 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:21.082 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:21.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:21.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:21.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:21.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:21.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:21.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:21.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:21.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:00:21.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:00:21.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:21.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:00:21.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:00:21.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:21.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:00:21.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:00:21.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:21.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:00:21.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:00:21.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:21.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:00:21.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:00:21.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:21.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:00:21.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:00:21.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:21.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:00:21.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:00:21.085 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:21.086 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:21.086 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:21.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.086 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:21.086 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:21.086 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 59.533828302000003, 2024-08-30T16:00:21.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:00:21.086 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:21.086 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:21.086 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:21.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:21.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:21.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:21.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:21.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:21.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:21.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:21.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:00:21.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:00:21.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:21.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:00:21.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:00:21.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:21.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:00:21.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:00:21.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:21.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:00:21.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:00:21.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:21.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:00:21.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:00:21.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:21.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:00:21.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:00:21.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:21.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:00:21.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:00:21.090 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:21.090 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:21.090 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:21.090 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:21.090 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:21.090 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:21.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:21.092 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:21.090+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:21.193 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:21.191+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:21.219 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:21.219 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:21.219 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:21.219 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:21.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:21.294 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:21.291+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:21.356 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:21.357 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:21.357 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:21.357 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:21.357 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:21.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:21.409 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:21.405+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:21.496 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:21.494+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:21.535 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:21.535 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:21.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:21.535 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:21.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.536 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:00:21.536 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:21.536 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 56.158084582999997, 2024-08-30T16:00:21.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:00:21.536 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:21.536 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:21.536 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:21.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:21.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:21.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:21.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:21.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:21.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:21.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:21.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:00:21.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:00:21.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:21.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:00:21.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:00:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:00:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:00:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:00:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:00:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:21.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:21.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:00:21.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:00:21.540 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:21.540 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:21.540 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:21.540 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:21.540 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:21.540 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:21.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:21.596 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:21.595+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:21.671 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:21.671 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:21.671 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:21.671 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:21.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:21.697 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:21.696+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:21.798 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:21.796+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:21.808 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:21.808 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:21.808 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:21.808 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:21.809 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:21.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:21.899 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:21.897+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:21.999 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:21.997+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:22.011 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:22.011 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:22.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:22.011 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:22.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.011 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:22.011 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:22.012 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.560473055000003, 2024-08-30T16:00:22.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:00:22.012 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:22.012 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:22.012 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:22.012 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:22.012 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:22.012 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:22.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.013 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:22.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:22.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:22.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:22.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:22.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:22.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:00:22.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:00:22.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:22.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:00:22.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:00:22.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:22.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:00:22.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:00:22.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:22.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:00:22.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:00:22.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:22.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:00:22.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:00:22.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:00:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:00:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:00:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:00:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:00:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:00:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:00:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:00:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:00:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:00:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:00:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:00:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:00:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:00:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:00:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:00:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:00:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:00:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:00:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 56.444606720000003, 2024-08-30T16:00:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:00:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:22.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:22.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:22.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:22.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:22.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:00:22.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:22.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:22.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:00:22.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:00:22.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:22.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:00:22.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:00:22.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:22.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:00:22.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:00:22.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:22.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:00:22.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:00:22.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:22.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:00:22.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:00:22.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:22.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:00:22.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:00:22.023 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.023 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:22.023 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.023 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:22.023 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:22.023 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 56.385015222, 2024-08-30T16:00:22.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:00:22.023 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:22.024 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:22.024 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:22.024 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:22.024 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:22.024 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:00:22.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.024 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:22.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:22.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:22.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:00:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:00:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:00:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:00:22.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:22.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:00:22.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:00:22.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:22.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:00:22.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:00:22.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:22.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:00:22.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:00:22.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:22.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:00:22.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:00:22.027 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.027 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:22.027 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.027 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:00:22.027 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:22.027 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 56.358316535, 2024-08-30T16:00:22.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:00:22.028 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:22.028 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:22.028 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:22.028 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:22.028 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:22.028 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:22.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.028 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:22.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:22.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:22.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:22.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:22.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:22.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:00:22.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:22.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:22.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:00:22.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:00:22.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:22.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:00:22.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:00:22.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:22.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:00:22.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:00:22.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:22.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:00:22.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:00:22.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:22.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:00:22.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:00:22.031 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.031 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:22.031 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.032 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:22.032 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:22.032 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.490700007000001, 2024-08-30T16:00:22.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:00:22.032 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:22.032 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:22.032 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:22.032 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:22.032 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:22.032 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:00:22.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.032 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:22.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:22.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:22.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:22.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:22.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:22.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:00:22.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:22.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:22.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:00:22.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:00:22.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:22.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:00:22.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:00:22.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:22.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:00:22.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:00:22.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:22.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:00:22.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:00:22.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:22.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:00:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:00:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:00:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:00:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:00:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:00:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:22.037 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.038 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:00:22.038 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:22.038 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.331452927999997, 2024-08-30T16:00:22.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:00:22.038 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:22.038 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:22.038 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:22.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:22.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:22.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:22.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:22.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:22.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:00:22.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:00:22.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:22.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:00:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:00:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:00:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:00:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:00:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:00:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:00:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:00:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:00:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:00:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:00:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:00:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.041 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:22.042 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.042 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:22.042 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:22.042 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.055278862000002, 2024-08-30T16:00:22.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:00:22.042 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:22.042 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:22.042 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:22.042 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:22.042 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:22.042 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:22.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:22.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:00:22.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:22.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:22.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:00:22.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:00:22.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:22.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:00:22.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:00:22.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:22.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:00:22.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:00:22.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:22.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:00:22.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:00:22.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:22.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:00:22.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:00:22.046 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.046 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:22.046 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.046 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:22.046 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:22.046 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 54.396987283000001, 2024-08-30T16:00:22.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:00:22.046 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:22.046 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:22.046 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:22.046 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:22.046 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:22.047 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:00:22.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.047 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:22.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:22.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:22.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:22.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:22.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:22.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:00:22.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:22.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:22.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:00:22.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:00:22.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:22.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:00:22.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:00:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:00:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:00:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:00:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:00:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:22.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:00:22.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:00:22.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:22.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:00:22.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:00:22.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:22.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:00:22.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:00:22.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:22.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:00:22.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:00:22.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:22.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:00:22.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:00:22.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:22.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:00:22.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:00:22.051 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.052 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:22.052 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.052 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:00:22.052 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:22.052 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 54.301073999000003, 2024-08-30T16:00:22.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:00:22.052 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:22.052 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:22.052 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:22.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:00:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:22.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:00:22.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:00:22.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:22.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:00:22.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:00:22.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:22.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:00:22.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:00:22.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:00:22.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.056 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:22.056 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.056 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:22.056 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:22.056 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:22.057 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:22.100 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:22.098+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:22.201 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:22.199+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:22.203 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:22.203 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:22.203 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:22.213 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:22.302 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:22.300+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:22.342 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:22.342 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:22.342 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:22.342 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:22.342 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:22.351 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:00:22.403 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:22.400+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:22.475 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:22.475 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:22.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:22.475 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:00:22.475 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:22.485 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:00:22.503 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:22.501+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:22.604 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:22.603+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:22.608 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:22.608 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:22.608 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:22.608 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:22.617 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:00:22.705 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:22.704+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:22.782 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:22.783 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:22.783 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:22.783 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:22.783 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:22.791 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:00:22.833 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:22.831+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:22.907 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:22.905+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:23.004 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:23.004 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:23.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 72.143307609000004, 2024-08-30T16:00:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:00:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:00:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:00:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:00:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:00:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:00:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:00:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:00:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:00:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:00:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:00:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:00:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:00:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:00:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:00:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:00:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:00:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:00:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:00:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:00:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:00:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:00:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:00:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:00:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:00:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.226891589000005, 2024-08-30T16:00:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:00:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:00:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:00:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:00:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:00:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:00:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:00:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:00:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:00:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:00:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:00:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:00:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:00:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:00:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:00:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:00:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:00:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:00:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:00:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 71.225187035999994, 2024-08-30T16:00:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:00:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:00:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:00:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:00:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:00:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:00:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:00:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:00:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:00:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:00:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:00:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:00:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:00:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:00:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:00:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.805733228999998, 2024-08-30T16:00:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:00:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:00:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:00:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:00:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:00:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:00:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:00:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:00:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:00:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:00:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:00:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:00:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:00:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.788845054999996, 2024-08-30T16:00:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:00:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:00:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:00:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:00:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:00:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:00:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:00:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:00:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:00:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:00:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:00:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:00:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:00:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:00:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:00:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:00:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:00:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.775844985999996, 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:00:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:00:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:00:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:00:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:00:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:00:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:00:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:00:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:00:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:00:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:00:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:00:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:00:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:00:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:00:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:00:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.773352693999996, 2024-08-30T16:00:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:00:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:00:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:00:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:00:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:00:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:00:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:00:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:00:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:00:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:00:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:00:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:00:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:00:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:00:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:00:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.772595245999995, 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:00:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:00:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:00:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:00:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:00:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:00:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:00:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:00:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:00:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:00:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:00:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:00:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:00:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:00:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.771862795999994, 2024-08-30T16:00:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:00:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:00:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:00:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:00:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:00:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:00:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:00:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:00:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:00:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.769323997000001, 2024-08-30T16:00:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:00:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:00:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:00:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:00:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:00:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:00:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:00:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:00:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:00:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:00:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:00:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 70.768575995999996, 2024-08-30T16:00:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:00:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:00:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:00:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:00:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:00:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:00:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:00:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:00:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.996459524000002, 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:00:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:00:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:00:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:00:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:00:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:00:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:00:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:00:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:00:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:00:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:00:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:00:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:00:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:00:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.994402378000004, 2024-08-30T16:00:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:00:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:00:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:00:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:00:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:00:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:00:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:00:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:00:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:00:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:00:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:00:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:00:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:00:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:00:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:00:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:00:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:00:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:00:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:00:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.731457878000001, 2024-08-30T16:00:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:00:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:00:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:00:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:00:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:00:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:00:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:00:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:00:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:00:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:00:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:00:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:00:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:00:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:00:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:00:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:00:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:00:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:00:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:00:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:00:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.730366375000003, 2024-08-30T16:00:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:00:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:00:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:00:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:00:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:00:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:00:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:00:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:00:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:00:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:00:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:00:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:00:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:00:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:00:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:00:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:00:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:00:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:00:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:00:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.729616789999994, 2024-08-30T16:00:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:00:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:00:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:00:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:00:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:00:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:00:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:00:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:00:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:00:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:00:23.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:00:23.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:00:23.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:00:23.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:00:23.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.088 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.088 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:23.088 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.695885258000004, 2024-08-30T16:00:23.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:00:23.088 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.088 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.088 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.088 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.088 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.088 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:00:23.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.088 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:23.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:23.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:00:23.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:00:23.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:00:23.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:00:23.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:00:23.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:00:23.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:00:23.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:00:23.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:00:23.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:00:23.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:00:23.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:00:23.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:00:23.093 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.093 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.093 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.093 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.093 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:23.093 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.695359249999996, 2024-08-30T16:00:23.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:00:23.093 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.093 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.094 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:00:23.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.094 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:23.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:23.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:00:23.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:23.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:00:23.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:00:23.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:00:23.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:00:23.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:00:23.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:00:23.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:00:23.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:00:23.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:00:23.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:00:23.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:00:23.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:00:23.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:00:23.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:00:23.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:00:23.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:00:23.099 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.099 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.099 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.099 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:23.099 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:23.099 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.694903686999993, 2024-08-30T16:00:23.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:00:23.099 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.099 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.099 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.099 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.100 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:00:23.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.100 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:23.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:23.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:00:23.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:00:23.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:00:23.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:00:23.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:00:23.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:00:23.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:00:23.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:00:23.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:00:23.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:00:23.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:00:23.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:00:23.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:00:23.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:00:23.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:00:23.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:00:23.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:00:23.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:00:23.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.104 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.104 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:00:23.104 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:23.104 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.024388371000001, 2024-08-30T16:00:23.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:00:23.105 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.105 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:23.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:23.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:23.105 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:00:23.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.105 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:23.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:23.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:00:23.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:23.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:00:23.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:00:23.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:00:23.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:00:23.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:00:23.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:00:23.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:00:23.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:00:23.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:00:23.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:00:23.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:23.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:00:23.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:00:23.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:00:23.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:00:23.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:00:23.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:00:23.109 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.109 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.110 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.110 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.110 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.110 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:23.111 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:00:23.112 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:23.109+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:23.212 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:23.210+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:23.276 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:23.276 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:23.276 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:23.276 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:23.286 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:00:23.330 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:23.328+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:23.412 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:23.415 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:23.413+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:23.421 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:00:23.516 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:23.514+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:23.589 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:23.589 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:23.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:23.589 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:23.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.590 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:00:23.590 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:23.590 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 57.059690248000003, 2024-08-30T16:00:23.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:00:23.590 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.590 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.590 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:23.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:23.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:00:23.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:00:23.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:00:23.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:00:23.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:00:23.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:00:23.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:00:23.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:00:23.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:00:23.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:00:23.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:00:23.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:00:23.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:00:23.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:00:23.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:00:23.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:00:23.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.594 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.595 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:00:23.595 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:23.595 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.941242101999997, 2024-08-30T16:00:23.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:00:23.595 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:23.595 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:23.595 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:23.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:23.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:23.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:23.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:23.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:23.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:23.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:00:23.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:00:23.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:23.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:00:23.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:00:23.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:23.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:00:23.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:00:23.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:23.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:00:23.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:00:23.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:23.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:00:23.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:00:23.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:23.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:00:23.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:00:23.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:23.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:00:23.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:00:23.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:23.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:23.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:23.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:00:23.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:00:23.599 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.599 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.599 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.599 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:23.599 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:23.599 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:23.600 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:00:23.617 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:23.615+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:23.718 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:23.717+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:23.724 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:23.724 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:23.724 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:23.724 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:23.733 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:00:23.819 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:23.817+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:23.898 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:23.898 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:23.898 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:23.898 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:23.899 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:23.908 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:00:23.920 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:23.918+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:24.021 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:24.019+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:24.072 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:24.072 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:24.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:24.072 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:24.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:24.073 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:24.073 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:24.073 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 55.694566483999999, 2024-08-30T16:00:24.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:00:24.073 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:24.073 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:00:24.073 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:24.073 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:00:24.073 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:24.073 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:00:24.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:24.074 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:24.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:24.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:24.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:24.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:24.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:24.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:24.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:24.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:24.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:24.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:24.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:24.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:00:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:00:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:00:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:00:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:00:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:00:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:24.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:24.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:00:24.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:00:24.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:24.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:24.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:24.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:00:24.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:00:24.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:24.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:24.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:24.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:00:24.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:00:24.076 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:24.077 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:24.077 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:24.077 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:24.077 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:24.077 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:24.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:00:24.126 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:24.123+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:24.222 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:24.219+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:24.222 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:24.222 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:24.223 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:24.223 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:24.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:00:24.323 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:24.320+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:24.402 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:24.402 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:24.402 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:24.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:00:24.452 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:24.450+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:24.526 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:24.523+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:24.585 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:24.585 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:24.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:24.585 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:00:24.585 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:24.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:24.626 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:24.625+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:24.750 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:24.748+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:24.763 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:24.764 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:24.764 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:24.764 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:24.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:24.857 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:24.855+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:24.932 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:24.930+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:24.940 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:24.940 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:24.940 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:24.940 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:24.941 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:24.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:25.033 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:25.032+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:25.127 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 63.851366032000001, 2024-08-30T16:00:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:00:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:00:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:00:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:00:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:00:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:00:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:00:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:00:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:00:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:00:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:00:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:00:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:00:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:00:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:00:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 63.578438671000001, 2024-08-30T16:00:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:00:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:00:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:00:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:00:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:00:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:25.139 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:25.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:25.141 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:25.138+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:25.241 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:25.240+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:25.307 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:25.307 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:25.307 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:25.308 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:25.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:25.360 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:25.358+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:25.443 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:25.443 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:25.443 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:25.443 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:25.443 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:25.444 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:25.441+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:25.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:25.544 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:25.542+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:25.626 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:25.627 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:25.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:25.627 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:25.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.627 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:00:25.627 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:25.627 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 60.249508853000002, 2024-08-30T16:00:25.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:00:25.627 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:25.627 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:25.628 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:25.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:25.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:25.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:25.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:25.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:25.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:25.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:25.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:00:25.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:00:25.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:25.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:00:25.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:00:25.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:25.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:00:25.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:00:25.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:25.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:00:25.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:00:25.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:25.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:00:25.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:00:25.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:25.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:00:25.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:00:25.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:25.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:00:25.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:00:25.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:25.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:25.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:25.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:00:25.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:00:25.632 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:25.632 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:25.632 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:25.632 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:25.632 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:25.632 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:25.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:25.645 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:25.644+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:25.746 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:25.744+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:25.805 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:25.805 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:25.805 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:25.805 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:25.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:25.858 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:25.856+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:25.941 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:25.941 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:25.941 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:25.941 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:25.942 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:25.948 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:25.947+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:25.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:26.049 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:26.047+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:26.140 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:26.140 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:26.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:26.140 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:26.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 61.691463868, 2024-08-30T16:00:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:00:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:00:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:00:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:00:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:00:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:00:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:00:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:00:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:00:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:00:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:00:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:00:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:00:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:00:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:26.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:00:26.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:00:26.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:26.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:00:26.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:00:26.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:00:26.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:00:26.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:00:26.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:00:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:00:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:00:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:00:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:00:26.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:00:26.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:26.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:00:26.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:00:26.147 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.147 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:26.147 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.147 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:00:26.147 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:26.147 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 60.575597533, 2024-08-30T16:00:26.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:00:26.148 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:26.148 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:26.148 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:26.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:26.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:26.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:26.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:26.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:26.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:00:26.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:26.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:26.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:00:26.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:00:26.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:26.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:00:26.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:00:26.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:26.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:00:26.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:00:26.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:26.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:00:26.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:00:26.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:26.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:00:26.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:00:26.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:26.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:00:26.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:00:26.151 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.151 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:26.151 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.151 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:26.151 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:26.151 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 60.516006034999997, 2024-08-30T16:00:26.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:00:26.152 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:26.152 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:26.152 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:26.152 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:26.152 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:26.152 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:00:26.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.152 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:26.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:26.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:26.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:00:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:00:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:00:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:00:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:00:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:00:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:00:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:00:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:26.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:00:26.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:00:26.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:26.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:00:26.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:00:26.155 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.155 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:26.155 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.155 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:00:26.155 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:26.156 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 60.489307347999997, 2024-08-30T16:00:26.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:00:26.156 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:26.156 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:26.156 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:26.156 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:26.156 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:26.156 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:26.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.156 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:26.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:26.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:26.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:26.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:26.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:26.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:00:26.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:26.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:26.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:00:26.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:00:26.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:26.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:00:26.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:00:26.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:26.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:00:26.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:00:26.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:26.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:00:26.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:00:26.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:26.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:00:26.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:00:26.164 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.164 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:26.164 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.165 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:26.165 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:26.165 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 59.621690819999998, 2024-08-30T16:00:26.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:00:26.165 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:26.165 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:26.165 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:26.165 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:26.165 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:26.165 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:00:26.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.165 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:26.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:26.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:26.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:26.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:26.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:26.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:00:26.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:26.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:26.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:00:26.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:00:26.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:26.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:00:26.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:00:26.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:26.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:00:26.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:00:26.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:26.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:00:26.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:00:26.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:26.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:00:26.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:00:26.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:26.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:00:26.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:00:26.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:26.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:00:26.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:00:26.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:26.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:00:26.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:00:26.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:26.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:00:26.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:00:26.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:26.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:00:26.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:00:26.171 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.171 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:26.171 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.171 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:00:26.171 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:26.171 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 59.462443741000001, 2024-08-30T16:00:26.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:00:26.171 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:26.171 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:26.171 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:26.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:00:26.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:00:26.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:26.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:00:26.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:00:26.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:26.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:00:26.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:00:26.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:26.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:00:26.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:00:26.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:26.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:00:26.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:00:26.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:26.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:00:26.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:00:26.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:26.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:00:26.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:00:26.175 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.175 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:26.175 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.175 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:26.175 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:26.175 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 59.186269674999998, 2024-08-30T16:00:26.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:00:26.176 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:26.176 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:26.176 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:26.176 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:26.176 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:26.176 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:26.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.176 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:26.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:26.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:26.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:26.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:26.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:26.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:00:26.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:26.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:26.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:00:26.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:00:26.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:26.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:00:26.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:00:26.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:26.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:00:26.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:00:26.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:26.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:00:26.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:00:26.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:26.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:00:26.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:00:26.179 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.179 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:26.179 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.180 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:26.180 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:26.180 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 58.527978095999998, 2024-08-30T16:00:26.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:00:26.180 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:26.180 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:26.180 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:26.180 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:26.180 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:26.180 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:00:26.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.181 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:26.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:26.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:26.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:26.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:26.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:26.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:00:26.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:26.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:26.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:00:26.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:00:26.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:26.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:00:26.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:00:26.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:26.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:00:26.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:00:26.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:26.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:00:26.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:00:26.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:26.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:00:26.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:00:26.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:26.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:00:26.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:00:26.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:26.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:00:26.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:00:26.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:26.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:00:26.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:00:26.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:26.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:00:26.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:00:26.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:26.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:00:26.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:00:26.185 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.186 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:26.186 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.186 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:00:26.186 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:26.186 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 58.432064812, 2024-08-30T16:00:26.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:00:26.186 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:26.186 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:26.186 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:26.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:26.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:00:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:26.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:00:26.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:00:26.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:26.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:00:26.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:00:26.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:26.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:00:26.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:00:26.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:26.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:00:26.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:00:26.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:26.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:00:26.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:00:26.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:26.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:26.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:26.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:00:26.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:00:26.190 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.190 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:26.190 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.190 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:26.190 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:26.190 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:26.192 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:26.192 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:26.190+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:26.298 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:26.294+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:26.341 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:26.341 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:26.341 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:26.350 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:26.395 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:26.393+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:26.480 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:26.480 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:26.480 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:26.480 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:26.480 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:26.490 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:00:26.495 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:26.493+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:26.596 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:26.594+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:26.671 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:26.671 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:26.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:26.671 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:00:26.672 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:26.681 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:00:26.723 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:26.721+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:26.799 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:26.797+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:26.848 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:26.848 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:26.848 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:26.848 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:26.857 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:00:26.900 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:26.897+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:26.983 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:26.983 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:26.983 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:26.983 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:26.983 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:26.993 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:00:27.037 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:27.035+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:27.103 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:27.101+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:27.113 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:00:27.114 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:00:27.164 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.164 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 76.303107952999994, 2024-08-30T16:00:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:00:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:00:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:00:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:00:27.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:00:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:00:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:00:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:00:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:00:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:00:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:00:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:00:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:00:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:00:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:00:27.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:00:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:00:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:00:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:00:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:00:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:00:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:00:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:00:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:00:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:00:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.170 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.386691932999994, 2024-08-30T16:00:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:00:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.171 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:00:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:00:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:00:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:00:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:00:27.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:00:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:00:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:00:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:00:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:00:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:00:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:00:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:00:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:00:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:00:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:00:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:00:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:00:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 75.384987379999998, 2024-08-30T16:00:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:00:27.176 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:00:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:00:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:00:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:00:27.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:00:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:00:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:00:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:00:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:00:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:00:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:00:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:00:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:00:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:00:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:00:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:00:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:00:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:00:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:00:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.181 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.965533573000002, 2024-08-30T16:00:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:00:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:00:27.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:00:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:00:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:00:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:00:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:00:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:00:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:00:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:00:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:00:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:00:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:00:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:00:27.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:00:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:00:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:00:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:00:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:00:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:00:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.948645399, 2024-08-30T16:00:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:00:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.187 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:00:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:00:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:00:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:00:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:00:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:00:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:00:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:00:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:00:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:00:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:00:27.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:00:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:00:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:00:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:00:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:00:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:00:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:00:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:00:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.192 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.93564533, 2024-08-30T16:00:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:00:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:00:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:00:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:00:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:00:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:00:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:00:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:00:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:00:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:00:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:00:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:00:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:00:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:00:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:00:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:00:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:00:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:00:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:00:27.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:00:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.933153038, 2024-08-30T16:00:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:00:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.198 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:00:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:27.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:00:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:00:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:00:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:00:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:00:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:00:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:00:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:00:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:00:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:00:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:00:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:00:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:00:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:00:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:00:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:00:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:00:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.203 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.932395589999999, 2024-08-30T16:00:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:00:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:00:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:27.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:00:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:00:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:00:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:00:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:00:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:00:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:00:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:00:27.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:00:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:00:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:00:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:00:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:00:27.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:00:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:00:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:00:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:00:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:00:27.208 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.931663139999998, 2024-08-30T16:00:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:00:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.209 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:00:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:00:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:00:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:00:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:00:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:00:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:00:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:00:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:00:27.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:00:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:00:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:00:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:00:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:00:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:00:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:00:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:00:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:00:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.214 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.929124341000005, 2024-08-30T16:00:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:00:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:00:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:00:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:00:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:00:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:00:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:00:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:00:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:00:27.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:00:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:00:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:00:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:00:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:00:27.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:00:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:00:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:00:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:00:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:00:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.219 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.92837634, 2024-08-30T16:00:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:00:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:00:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.220 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:00:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:00:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:00:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:00:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:00:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:00:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:00:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:00:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:00:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:00:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:00:27.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:00:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:00:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:00:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:00:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:00:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:00:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:00:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.156259868000006, 2024-08-30T16:00:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:00:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:00:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:00:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:00:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:00:27.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:00:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:00:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:00:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:00:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:00:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:00:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:00:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:00:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:00:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:00:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:00:27.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:00:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:00:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:00:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:00:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.230 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 74.154202721999994, 2024-08-30T16:00:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:00:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:00:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.231 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:00:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:00:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:00:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:00:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:00:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:00:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:00:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:00:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:00:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:00:27.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:00:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:00:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:00:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:00:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:00:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:00:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:00:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 73.891258222000005, 2024-08-30T16:00:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:00:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.236 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:00:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:00:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:00:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:00:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:00:27.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:00:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:00:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:00:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:00:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:00:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:00:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:00:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:00:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:00:27.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:00:27.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:00:27.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:00:27.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:00:27.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:00:27.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.242 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.242 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.242 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:27.242 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 73.890166719000007, 2024-08-30T16:00:27.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:00:27.242 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.242 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.243 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.243 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.243 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.243 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:00:27.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.243 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:27.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:27.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:00:27.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:27.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:00:27.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:00:27.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:00:27.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:00:27.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:00:27.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:00:27.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:00:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:00:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:00:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:00:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:00:27.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:00:27.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:00:27.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:00:27.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:00:27.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:00:27.247 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.247 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.248 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.248 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.248 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:27.248 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 73.889417133999999, 2024-08-30T16:00:27.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:00:27.248 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.248 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.248 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.248 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.248 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.248 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:00:27.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.249 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:27.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:27.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:00:27.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:27.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:00:27.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:00:27.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:00:27.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:00:27.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:00:27.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:00:27.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:00:27.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:00:27.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:00:27.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:00:27.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:00:27.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:00:27.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:00:27.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:00:27.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:00:27.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:00:27.253 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.253 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.253 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.254 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.254 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:27.254 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 73.855685601999994, 2024-08-30T16:00:27.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:00:27.254 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.254 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.254 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.254 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.254 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.254 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:00:27.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.254 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:27.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:27.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:00:27.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:00:27.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:00:27.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:00:27.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:00:27.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:00:27.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:00:27.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:00:27.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:00:27.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:00:27.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:00:27.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:00:27.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:00:27.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:00:27.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:00:27.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:00:27.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:00:27.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:00:27.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.259 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.259 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.259 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:27.259 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 73.855159594, 2024-08-30T16:00:27.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:00:27.259 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.259 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.260 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:00:27.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.260 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:27.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:27.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:00:27.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:27.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:00:27.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:00:27.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:00:27.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:00:27.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:00:27.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:00:27.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:00:27.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:00:27.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:00:27.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:00:27.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:00:27.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:00:27.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:00:27.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:00:27.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:00:27.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:00:27.264 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.264 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.264 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.264 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:27.264 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:27.265 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 73.854704030999997, 2024-08-30T16:00:27.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:00:27.265 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.265 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.265 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.265 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.265 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.265 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:00:27.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.265 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:27.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:27.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:00:27.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:00:27.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:00:27.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:00:27.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:00:27.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:00:27.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:00:27.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:00:27.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:00:27.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:00:27.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:00:27.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:00:27.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:00:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:00:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:00:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:00:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:00:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:00:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:00:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.184188715000005, 2024-08-30T16:00:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:00:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:27.271 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:00:27.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.271 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:27.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:27.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:27.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:27.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:27.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:27.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:27.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:00:27.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:27.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:27.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:00:27.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:00:27.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:27.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:00:27.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:00:27.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:27.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:00:27.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:00:27.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:27.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:00:27.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:00:27.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:27.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:00:27.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:00:27.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:27.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:00:27.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:00:27.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:27.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:00:27.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:00:27.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:27.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:27.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:27.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:00:27.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:00:27.275 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.275 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.275 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.275 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:27.275 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:27.275 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:27.276 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:00:27.277 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:00:27.278 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:00:27.278 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":16,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:00:27.279 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2024-08-30T16:00:27.279 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-30T16:00:27.279 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-08-30T16:00:27.279 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:27.275+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:27.320 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:00:27.320 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:00:27.321 INFO:tasks.ceph.mon.c:Started 2024-08-30T16:00:27.322 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-08-30T16:00:27.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:00:27.378 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:27.376+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:27.422 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:00:27.422 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:00:27.452 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:27.452 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:27.452 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:27.452 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:27.461 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:00:27.478 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:27.477+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:27.517 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:00:27.518 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:00:27.602 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:00:27.603 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:00:27.603 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:27.601+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:27.666 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:27.667 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:27.667 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:27.667 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:27.667 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:27.675 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:00:27.717 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:27.715+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:27.725 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:00:27.725 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:00:27.781 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:27.779+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:27.810 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:00:27.810 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:00:27.843 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:27.843 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:27.852 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:00:27.895 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:27.892+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:27.983 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:27.982+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:28.000 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":16,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:00:28.023 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:28.023 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:28.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:00:28.085 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-08-30T16:00:28.087 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:28.086+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:28.088 INFO:teuthology.orchestra.run.smithi050.stderr:set 2.3 pg_upmap mapping to [7,6] 2024-08-30T16:00:28.180 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:28.180 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:28.185 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:28.184+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:28.189 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:00:28.287 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:28.285+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:28.388 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:28.386+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:28.465 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:28.465 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:28.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:00:28.518 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:28.517+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:28.589 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:28.587+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:28.622 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:28.623 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:28.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:00:28.717 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:28.715+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:28.790 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:28.789+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:28.814 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:28.814 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:28.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:00:28.891 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:28.890+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:28.992 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:28.990+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:28.996 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:28.997 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:29.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:29.093 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:29.091+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:29.166 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:29.166 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:29.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:29.216 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:29.215+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:29.294 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:29.293+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:29.359 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:29.359 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:29.369 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:29.394 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:29.393+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:29.496 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:29.495+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:29.537 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:29.538 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:29.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:29.631 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:29.630+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:29.699 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:29.697+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:29.734 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:29.734 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:29.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:29.799 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:29.797+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:29.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:00:29.944 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:29.945 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:29.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:30.001 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:29.999+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:30.013 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:30.000+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:30.081 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:00:30.096 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:00:30.096 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:30.096 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:30.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-08-30T16:00:30.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:30.144 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:30.142+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:30.186 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:00:30.186 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:00:30.227 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:30.226+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:30.279 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:00:30.280 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:00:30.305 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:30.303+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:30.360 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:30.360 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:30.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:30.413 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:30.412+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:30.529 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:30.527+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:30.629 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:00:30.629 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:00:30.630 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:30.629+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:30.631 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:30.631 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:30.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:30.710 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:00:30.710 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:00:30.711 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:30.709+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:30.799 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:00:30.800 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:00:30.809 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:30.808+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:30.838 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:30.839 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:30.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:30.839 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:30.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.839 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:30.840 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:30.840 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 66.387579122999995, 2024-08-30T16:00:30.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:00:30.840 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:30.840 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:30.840 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:30.841 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:30.841 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:30.841 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:30.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.841 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:30.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:30.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:30.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:30.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:30.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:30.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:00:30.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:00:30.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:00:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:00:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:00:30.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:00:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:00:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:00:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:00:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:00:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:00:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:00:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:00:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:00:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:00:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:00:30.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:00:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:00:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:00:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:00:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:00:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:00:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:00:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:00:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:00:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:00:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:00:30.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:00:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:00:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 65.271712788000002, 2024-08-30T16:00:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:00:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:30.850 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:00:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:00:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:00:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:00:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:00:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:00:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:00:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:00:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:00:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:00:30.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:00:30.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:30.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:00:30.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:00:30.854 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.854 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:30.854 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.854 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:30.854 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:30.855 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 65.212121289999999, 2024-08-30T16:00:30.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:00:30.855 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:30.855 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:30.855 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:30.855 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:30.855 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:30.855 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:00:30.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.855 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:30.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:30.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:30.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:30.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:30.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:30.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:00:30.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:00:30.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:30.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:00:30.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:00:30.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:30.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:00:30.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:00:30.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:30.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:00:30.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:00:30.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:30.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:00:30.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:00:30.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:30.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:00:30.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:00:30.858 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.858 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:30.858 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.859 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:00:30.859 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:30.859 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 65.185422603000006, 2024-08-30T16:00:30.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:00:30.859 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:30.859 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:30.859 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:30.859 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:30.859 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:30.859 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:30.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.860 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:30.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:30.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:30.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:30.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:30.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:30.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:00:30.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:30.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:30.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:00:30.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:00:30.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:30.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:00:30.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:00:30.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:30.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:00:30.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:00:30.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:30.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:00:30.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:00:30.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:30.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:00:30.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:00:30.863 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.863 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:30.863 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.863 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:30.863 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:30.863 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 64.317806074999993, 2024-08-30T16:00:30.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:00:30.863 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:30.864 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:30.864 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:30.864 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:30.864 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:30.864 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:00:30.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.864 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:30.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:30.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:30.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:30.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:30.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:30.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:00:30.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:30.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:30.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:00:30.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:00:30.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:30.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:00:30.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:00:30.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:30.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:00:30.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:00:30.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:30.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:00:30.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:00:30.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:30.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:00:30.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:00:30.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:30.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:00:30.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:00:30.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:30.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:00:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:00:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:00:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:00:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:00:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:00:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:30.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:00:30.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:00:30.871 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.871 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:30.871 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.871 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:00:30.871 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:30.871 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 64.158558995999996, 2024-08-30T16:00:30.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:00:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:30.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:00:30.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:00:30.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:30.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:00:30.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:00:30.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:30.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:00:30.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:00:30.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:30.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:00:30.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:00:30.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:30.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:00:30.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:00:30.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:30.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:00:30.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:00:30.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:30.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:00:30.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:00:30.875 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.876 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:30.876 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.876 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:30.876 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:30.876 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 63.882384930000001, 2024-08-30T16:00:30.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:00:30.876 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:30.876 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:30.876 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:30.876 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:30.876 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:30.877 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:30.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.877 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:30.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:30.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:30.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:30.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:30.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:30.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:00:30.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:30.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:30.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:00:30.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:00:30.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:30.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:00:30.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:00:30.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:30.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:00:30.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:00:30.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:30.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:00:30.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:00:30.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:30.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:00:30.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:00:30.881 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.881 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:30.881 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.881 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:30.881 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:30.881 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 63.224093351, 2024-08-30T16:00:30.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:00:30.881 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:30.881 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:30.881 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:30.882 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:30.882 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:30.882 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:00:30.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.882 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:30.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:30.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:30.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:30.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:30.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:30.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:00:30.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:30.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:30.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:00:30.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:00:30.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:30.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:00:30.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:00:30.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:30.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:00:30.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:00:30.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:30.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:00:30.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:00:30.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:30.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:00:30.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:00:30.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:30.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:00:30.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:00:30.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:30.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:00:30.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:00:30.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:30.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:00:30.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:00:30.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:30.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:00:30.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:00:30.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:30.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:00:30.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:00:30.889 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.889 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:30.889 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.889 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:00:30.889 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:30.889 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 63.128180067000002, 2024-08-30T16:00:30.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:00:30.889 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:30.889 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:30.889 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:30.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:30.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:30.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:30.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:30.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:30.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:00:30.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:00:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:00:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:00:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:00:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:00:30.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:00:30.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:30.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:00:30.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:00:30.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:30.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:00:30.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:00:30.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:30.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:00:30.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:00:30.893 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.893 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:30.894 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.894 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:30.894 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:30.894 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.8768195560000001, 2024-08-30T16:00:30.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:00:30.894 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:30.895 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:30.895 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:30.895 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:30.895 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:30.895 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:30.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.895 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:30.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:30.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:30.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:30.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:30.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:30.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:30.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:00:30.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:00:30.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:30.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:00:30.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:00:30.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:30.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:00:30.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:00:30.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:30.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:00:30.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:00:30.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:30.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:00:30.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:00:30.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:30.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:30.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:30.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:00:30.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:00:30.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.902 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:30.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:30.902 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:30.902 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:30.903 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:30.904 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:00:30.904 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:00:30.944 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:30.943+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:30.969 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:00:30.969 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:00:31.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:00:31.023 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:31.021+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:31.033 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:31.034 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:31.034 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:31.034 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:31.044 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:31.112 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:31.110+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:31.186 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:31.186 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:31.187 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:31.187 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:31.187 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:31.197 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:00:31.239 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:31.237+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:31.313 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:31.311+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:31.363 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:31.363 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:31.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:31.363 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:00:31.364 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:31.372 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:00:31.417 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:31.415+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:31.499 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-08-30T16:00:31.525 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:31.525 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:31.525 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:31.525 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:31.534 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:00:31.555 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:31.554+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:31.619 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:31.617+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:31.698 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:31.698 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:31.698 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:31.708 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:00:31.750 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:31.748+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:31.754 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":16,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:00:31.818 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:31.816+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:31.919 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:31.917+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:31.920 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:31.920 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:31.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:31.921 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:31.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.921 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:31.921 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:31.921 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 81.059659284999995, 2024-08-30T16:00:31.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:00:31.921 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:31.921 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:31.921 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:31.921 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:31.922 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:31.922 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:00:31.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.922 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:31.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:31.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:31.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:31.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:31.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:31.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:00:31.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:00:31.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.923 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:31.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:00:31.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:00:31.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:31.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:00:31.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:00:31.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:31.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:00:31.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:00:31.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:31.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:00:31.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:00:31.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:00:31.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:00:31.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:00:31.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:31.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:00:31.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:00:31.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:31.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:00:31.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:00:31.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:31.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:00:31.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:00:31.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:31.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:00:31.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:00:31.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:31.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:00:31.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:00:31.927 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.927 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:31.927 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.928 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:31.928 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:31.928 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 80.143243264999995, 2024-08-30T16:00:31.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:00:31.928 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:31.928 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:31.928 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:31.928 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:31.928 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:31.928 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:00:31.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.929 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:31.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:31.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:31.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:31.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:31.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:31.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:00:31.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:31.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:31.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:00:31.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:00:31.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:31.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:00:31.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:00:31.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:31.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:00:31.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:00:31.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:31.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:00:31.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:00:31.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:31.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:00:31.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:00:31.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:31.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:00:31.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:00:31.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:31.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:00:31.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:00:31.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:31.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:00:31.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:00:31.933 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.933 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:31.933 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.933 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:31.933 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:31.933 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 80.141538711999999, 2024-08-30T16:00:31.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:00:31.933 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:31.933 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:31.934 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:31.934 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:31.934 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:31.934 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:00:31.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.934 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:31.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:31.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:31.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:31.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:31.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:31.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:00:31.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:00:31.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:31.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:00:31.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:00:31.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:31.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:00:31.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:00:31.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:31.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:00:31.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:00:31.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:31.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:00:31.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:00:31.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:31.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:00:31.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:00:31.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:31.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:00:31.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:00:31.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:31.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:00:31.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:00:31.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:31.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:00:31.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:00:31.939 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.939 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:31.939 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.939 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:31.939 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:31.939 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.722084905000003, 2024-08-30T16:00:31.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:00:31.939 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:31.940 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:31.940 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:31.940 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:31.940 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:31.940 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:00:31.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.940 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:31.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:31.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:31.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:31.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:31.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:31.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:00:31.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:00:31.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:31.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:00:31.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:00:31.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:31.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:00:31.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:00:31.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:31.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:00:31.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:00:31.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:31.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:00:31.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:00:31.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:31.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:00:31.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:00:31.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:31.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:00:31.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:00:31.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:31.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:00:31.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:00:31.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:31.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:00:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:00:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.705196731000001, 2024-08-30T16:00:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:00:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:31.945 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:31.946 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:31.946 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:31.946 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:31.946 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:00:31.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.946 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:31.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:31.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:31.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:31.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:31.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:31.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:00:31.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:00:31.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:31.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:00:31.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:00:31.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:31.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:00:31.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:00:31.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:31.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:00:31.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:00:31.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:31.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:00:31.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:00:31.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:31.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:00:31.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:00:31.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:31.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:00:31.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:00:31.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:31.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:00:31.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:00:31.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:31.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:00:31.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:00:31.951 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.951 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:31.951 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.951 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:31.951 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:31.951 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.692196662000001, 2024-08-30T16:00:31.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:00:31.952 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:31.952 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:31.952 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:31.952 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:31.952 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:31.952 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:00:31.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.952 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:31.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:31.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:31.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:31.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:31.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:31.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:00:31.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:00:31.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:31.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:00:31.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:00:31.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:31.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:00:31.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:00:31.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:31.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:00:31.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:00:31.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:31.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:00:31.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:00:31.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:31.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:00:31.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:00:31.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:31.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:00:31.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:00:31.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:31.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:00:31.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:00:31.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:31.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:00:31.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:00:31.957 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.957 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:31.957 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.957 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:31.957 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:31.957 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.689704370000001, 2024-08-30T16:00:31.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:00:31.958 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:31.958 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:31.958 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:31.958 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:31.958 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:31.958 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:00:31.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.958 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:31.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:31.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:31.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:31.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:31.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:31.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:00:31.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:31.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:31.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:00:31.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:00:31.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:31.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:00:31.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:00:31.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:31.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:00:31.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:00:31.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:31.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:00:31.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:00:31.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:31.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:00:31.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:00:31.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:31.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:00:31.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:00:31.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:31.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:00:31.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:00:31.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:31.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:00:31.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:00:31.962 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.962 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:31.963 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.963 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:31.963 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:31.963 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.688946922, 2024-08-30T16:00:31.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:00:31.963 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:31.963 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:31.963 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:31.963 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:31.963 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:31.964 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:00:31.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.964 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:31.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:31.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:31.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:31.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:31.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:31.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:00:31.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:00:31.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:31.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:00:31.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:00:31.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:31.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:00:31.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:00:31.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:31.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:00:31.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:00:31.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:31.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:00:31.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:00:31.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:31.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:00:31.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:00:31.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:31.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:00:31.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:00:31.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:31.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:00:31.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:00:31.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:31.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:00:31.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:00:31.968 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.968 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:31.968 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.969 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:31.969 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:31.969 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.688214471999999, 2024-08-30T16:00:31.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:00:31.969 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:31.969 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:31.969 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:31.969 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:31.969 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:31.969 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:00:31.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.969 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:31.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:31.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:31.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:31.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:31.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:31.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:00:31.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:31.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:31.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:00:31.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:00:31.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:31.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:00:31.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:00:31.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:31.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:00:31.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:00:31.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:31.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:00:31.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:00:31.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:31.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:00:31.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:00:31.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:31.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:00:31.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:00:31.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:31.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:00:31.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:00:31.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:31.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:00:31.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:00:31.974 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.974 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:31.975 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.975 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:31.975 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:31.975 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.685675673000006, 2024-08-30T16:00:31.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:00:31.975 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:31.975 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:31.975 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:31.975 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:31.975 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:31.975 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:00:31.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.976 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:31.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:31.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:31.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:31.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:31.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:31.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:00:31.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:31.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:31.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:00:31.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:00:31.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:31.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:00:31.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:00:31.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:31.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:00:31.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:00:31.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:31.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:00:31.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:00:31.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:31.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:00:31.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:00:31.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:31.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:00:31.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:00:31.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:31.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:00:31.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:00:31.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:31.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:00:31.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:00:31.981 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.981 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:31.981 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.981 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:31.981 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:31.981 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 79.684927672000001, 2024-08-30T16:00:31.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:00:31.981 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:31.981 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:31.981 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:31.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:31.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:31.982 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:00:31.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.982 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:31.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:31.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:31.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:31.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:31.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:31.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:00:31.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:00:31.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:31.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:00:31.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:00:31.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:31.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:00:31.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:00:31.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:31.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:00:31.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:00:31.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:31.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:00:31.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:00:31.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:31.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:00:31.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:00:31.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:31.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:00:31.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:00:31.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:31.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:00:31.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:00:31.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:31.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:00:31.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:00:31.987 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.987 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:31.987 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.987 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:31.987 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:31.987 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.912811199999993, 2024-08-30T16:00:31.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:00:31.988 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:31.988 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:31.988 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:31.988 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:31.988 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:31.988 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:00:31.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.988 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:31.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:31.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:31.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:31.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:31.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:31.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:00:31.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:00:31.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:00:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:00:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:00:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:00:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:31.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:00:31.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:00:31.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:31.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:00:31.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:00:31.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:31.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:00:31.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:00:31.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:31.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:00:31.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:00:31.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:31.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:00:31.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:00:31.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:00:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:00:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.910754053999995, 2024-08-30T16:00:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:00:31.993 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:31.994 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:31.994 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:31.994 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:31.994 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:31.994 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:00:31.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.994 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:31.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:31.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:31.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:31.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:31.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:31.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:31.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:00:31.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:31.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:31.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:00:31.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:00:31.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:31.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:00:31.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:00:31.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:31.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:00:31.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:00:31.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:31.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:00:31.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:00:31.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:31.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:00:31.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:00:31.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:31.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:00:31.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:00:31.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:31.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:00:31.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:00:31.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:31.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:00:31.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:00:31.998 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.998 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:31.999 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:31.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:31.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:31.999 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:31.999 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:31.999 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.647809554000006, 2024-08-30T16:00:31.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:00:31.999 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:31.999 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:31.999 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:31.999 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:32.000 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:32.000 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:00:32.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.000 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:32.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:32.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:32.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:32.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:32.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:32.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:00:32.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:00:32.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:32.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:00:32.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:00:32.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:32.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:00:32.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:00:32.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:32.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:00:32.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:00:32.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:32.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:00:32.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:00:32.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:32.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:00:32.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:00:32.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:32.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:00:32.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:00:32.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:32.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:00:32.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:00:32.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:32.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:00:32.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:00:32.004 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.004 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:32.004 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.005 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:32.005 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:32.005 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.646718050999993, 2024-08-30T16:00:32.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:00:32.005 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:32.005 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:32.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:32.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:32.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:32.005 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:00:32.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.005 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:32.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:32.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:32.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:32.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:32.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:32.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:00:32.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:32.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:32.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:00:32.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:00:32.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:32.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:00:32.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:00:32.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:32.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:00:32.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:00:32.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:32.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:00:32.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:00:32.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:32.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:00:32.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:00:32.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:32.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:00:32.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:00:32.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:32.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:00:32.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:00:32.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:32.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:00:32.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:00:32.010 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.010 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:32.010 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.011 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:32.011 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:32.011 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.645968465999999, 2024-08-30T16:00:32.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:00:32.011 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:32.011 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:32.011 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:32.011 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:32.011 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:32.012 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:00:32.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.012 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:32.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:32.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:32.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:32.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:32.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:32.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:00:32.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:32.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:32.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:00:32.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:00:32.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:32.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:00:32.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:00:32.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:32.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:00:32.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:00:32.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:32.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:00:32.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:00:32.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:32.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:00:32.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:00:32.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:32.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:00:32.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:00:32.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:32.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:00:32.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:00:32.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:32.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:00:32.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:00:32.016 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.016 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:32.016 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.016 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:32.016 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:32.017 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.612236933999995, 2024-08-30T16:00:32.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:00:32.017 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:32.017 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:32.017 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:32.017 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:32.017 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:32.017 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:00:32.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.017 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:32.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:32.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:32.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:32.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:32.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:32.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:00:32.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:00:32.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:32.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:00:32.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:00:32.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:32.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:00:32.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:00:32.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:32.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:00:32.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:00:32.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:32.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:00:32.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:00:32.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:32.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:00:32.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:00:32.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:32.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:00:32.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:00:32.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:32.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:00:32.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:00:32.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:32.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:00:32.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:00:32.022 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.022 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:32.022 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.023 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:32.023 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:32.023 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.611710926000001, 2024-08-30T16:00:32.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:00:32.023 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:32.023 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:32.023 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:32.023 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:32.023 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:32.024 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:00:32.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.024 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:32.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:32.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:32.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:32.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:32.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:32.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:00:32.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:32.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:32.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:00:32.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:00:32.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:32.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:00:32.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:00:32.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:32.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:00:32.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:00:32.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:32.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:00:32.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:00:32.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:32.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:00:32.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:00:32.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:32.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:00:32.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:00:32.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:32.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:00:32.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:00:32.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:32.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:00:32.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:00:32.028 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.028 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:32.029 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.029 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:32.029 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:32.029 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.611255362999998, 2024-08-30T16:00:32.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:00:32.029 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:32.029 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:32.029 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:32.029 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:32.029 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:32.029 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:00:32.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.030 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:32.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:32.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:32.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:32.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:32.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:32.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:00:32.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:00:32.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:32.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:00:32.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:00:32.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:32.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:00:32.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:00:32.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:32.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:00:32.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:00:32.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:32.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:00:32.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:00:32.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:32.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:00:32.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:00:32.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:32.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:00:32.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:00:32.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:32.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:00:32.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:00:32.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:32.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:00:32.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:00:32.034 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.034 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:32.034 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.034 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:00:32.034 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:32.034 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 73.940740047000006, 2024-08-30T16:00:32.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:00:32.035 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:32.035 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:32.035 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:32.035 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:32.035 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:32.035 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:00:32.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.035 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:32.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:32.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:32.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:32.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:32.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:32.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:00:32.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:32.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:32.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:00:32.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:00:32.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:32.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:00:32.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:00:32.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:32.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:00:32.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:00:32.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:32.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:00:32.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:00:32.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:32.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:00:32.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:00:32.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:32.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:00:32.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:00:32.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:32.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:00:32.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:00:32.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:32.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:00:32.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:00:32.039 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.040 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:32.040 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.040 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.040 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:32.040 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:32.041 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:00:32.042 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:32.040+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:32.143 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:32.142+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:32.167 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:32.167 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:32.167 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:32.167 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:32.177 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:00:32.244 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:32.242+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:32.308 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:32.309 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:32.309 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:32.309 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:32.309 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:32.318 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:00:32.360 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:32.358+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:32.446 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:32.445+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:32.490 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:32.490 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:32.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:32.490 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:32.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.490 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:00:32.490 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:32.490 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 65.960129092000003, 2024-08-30T16:00:32.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:00:32.490 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:32.490 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:32.491 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:32.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:32.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:32.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:32.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:32.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:32.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:00:32.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:00:32.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:32.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:00:32.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:00:32.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:32.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:00:32.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:00:32.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:32.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:00:32.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:00:32.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:32.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:00:32.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:00:32.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:32.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:00:32.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:00:32.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:32.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:00:32.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:00:32.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:32.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:00:32.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:00:32.494 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.494 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:32.494 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.495 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:00:32.495 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:32.495 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 64.841680945999997, 2024-08-30T16:00:32.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:00:32.495 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:32.495 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:32.495 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:32.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:32.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:32.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:32.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:32.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:32.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:00:32.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:00:32.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:32.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:00:32.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:00:32.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:32.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:00:32.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:00:32.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:32.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:00:32.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:00:32.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:32.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:00:32.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:00:32.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:32.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:00:32.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:00:32.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:32.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:00:32.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:00:32.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:32.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:00:32.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:00:32.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.499 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:32.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.499 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:32.499 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:32.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:00:32.547 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:32.546+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:32.648 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:32.647+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:32.670 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:32.670 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:32.670 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:32.671 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:32.680 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:00:32.751 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:32.749+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:32.806 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:32.806 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:32.806 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:32.806 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:32.807 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:32.815 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:00:32.852 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:32.850+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:32.940 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:32.940 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:32.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:32.940 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:32.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.940 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:32.940 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:32.941 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 64.562207275999995, 2024-08-30T16:00:32.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:00:32.941 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:32.941 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:00:32.941 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:32.941 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:00:32.941 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:32.941 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:00:32.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.941 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:32.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:32.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:32.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:32.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:32.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:32.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:32.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:00:32.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:00:32.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:32.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:00:32.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:00:32.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:32.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:00:32.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:00:32.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:32.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:00:32.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:00:32.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:32.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:00:32.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:00:32.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:32.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:32.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:32.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:00:32.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:00:32.945 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.945 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:32.945 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.945 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:32.945 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:32.945 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:32.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:00:32.952 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:32.951+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:32.982 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:00:33.053 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:33.052+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:33.078 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:33.079 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:33.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:00:33.103 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:00:33.103 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:00:33.103 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-08-30T16:00:33.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-08-30T16:00:33.154 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:33.152+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:33.254 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:33.253+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:33.270 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:00:33.270 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:00:33.324 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:33.324 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:33.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:00:33.355 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:33.354+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:33.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-08-30T16:00:33.373 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:00:33.373 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:00:33.457 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:33.455+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:33.485 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:00:33.485 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:00:33.492 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:33.492 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:33.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:33.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-08-30T16:00:33.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-08-30T16:00:33.597 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:00:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:00:33.598 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:33.597+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:33.658 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:33.657+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:33.684 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:33.684 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:33.687 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:00:33.687 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:00:33.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:33.782 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:33.780+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:33.782 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:00:33.782 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:00:33.834 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:33.834 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:33.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:33.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-08-30T16:00:33.860 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:33.858+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:33.867 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:00:33.868 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:00:33.963 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:00:33.963 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:00:33.973 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:33.972+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:33.989 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:33.989 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:34.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:34.061 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:34.060+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:34.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-08-30T16:00:34.163 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:34.161+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:34.189 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:34.190 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:34.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:34.279 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:34.277+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:34.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-08-30T16:00:34.365 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:34.363+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:34.395 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:34.395 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:34.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:34.465 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:34.463+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:34.525 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-08-30T16:00:34.569 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:34.567+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:34.583 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:34.583 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:34.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:34.625 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:35.089 30228== Warning: unimplemented fcntl command: 1036 2024-08-30T16:00:34.657 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:35.929 22555== Warning: unimplemented fcntl command: 1036 2024-08-30T16:00:34.659 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:35.931 22555== Warning: unimplemented fcntl command: 1036 2024-08-30T16:00:34.666 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:34.665+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:34.691 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:35.964 22555== Warning: unimplemented fcntl command: 1036 2024-08-30T16:00:34.699 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:35.164 30228== Warning: unimplemented fcntl command: 1036 2024-08-30T16:00:34.766 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:34.767 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:34.767 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:34.766+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:34.769 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:00:34.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:34.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-08-30T16:00:34.869 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:34.867+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:34.969 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:34.968+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:34.991 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:34.992 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:35.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:35.069 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:35.068+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:35.131 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:35.596 30228== Warning: unimplemented fcntl command: 1036 2024-08-30T16:00:35.171 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:35.170+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:35.182 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:35.182 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:35.192 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:35.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-08-30T16:00:35.271 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:35.270+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:35.321 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:35.321 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:35.330 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:35.379 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:35.378+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:35.414 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":16,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:00:35.473 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:35.472+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:35.507 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:35.507 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:35.517 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:00:35.577 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:35.575+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:35.646 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:08.263 33106== Warning: unimplemented fcntl command: 1036 2024-08-30T16:00:35.654 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:35.654 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:35.662 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:00:35.704 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:35.703+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:35.780 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:35.776+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:35.828 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:35.828 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:35.837 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:00:35.879 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:35.878+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:35.896 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:08.512 33106== Warning: unimplemented fcntl command: 1036 2024-08-30T16:00:35.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:00:35.980 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:35.979+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:36.002 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:36.002 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:36.011 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:00:36.081 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:36.079+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:36.135 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:36.136 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:36.145 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:00:36.189 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:36.188+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:36.254 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:00:36.255 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:00:36.282 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:36.281+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:36.314 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:36.314 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:36.323 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:00:36.383 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:36.382+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:36.385 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:00:36.385 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:00:36.474 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:00:36.474 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:00:36.484 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:36.482+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:36.488 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:36.489 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:36.498 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:00:36.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-08-30T16:00:36.582 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:00:36.582 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:00:36.584 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:36.582+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:36.626 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:36.626 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:36.635 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:00:36.685 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:36.684+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:36.687 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:00:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:00:36.762 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:36.762 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:36.772 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:00:36.789 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:00:36.789 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:00:36.817 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:36.815+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:36.887 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:36.886+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:36.888 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:00:36.888 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:00:36.913 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:36.914 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:36.923 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:00:36.987 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:00:36.987 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:00:37.007 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:37.005+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:37.088 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 68.709830013000001, 2024-08-30T16:00:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:00:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:00:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:00:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:00:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:00:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:00:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:00:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:00:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:00:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:00:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:00:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:00:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:00:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:00:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:00:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:00:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:37.093 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:37.093 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:37.093 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:37.087+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:37.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:00:37.190 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:37.189+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:37.225 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:37.225 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:37.225 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:37.225 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:37.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:00:37.291 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:37.290+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:37.393 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:37.391+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:37.404 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:37.404 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:37.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:00:37.493 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:37.492+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:37.542 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:37.542 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:37.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:37.542 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:00:37.542 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:37.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:37.594 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:37.592+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:37.695 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:37.693+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:37.729 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-08-30T16:00:37.736 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:37.736 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:37.736 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:37.736 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:37.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:37.795 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:37.793+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:37.896 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:37.895+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:37.927 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:37.927 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:37.927 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:37.927 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:37.927 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:37.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:37.997 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:37.996+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:38.099 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:38.097+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:38.128 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:38.128 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:38.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:38.128 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:38.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.128 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:38.128 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:38.128 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 76.853403380000003, 2024-08-30T16:00:38.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:00:38.129 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:38.129 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:38.129 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:38.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:38.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:38.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:38.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:38.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:38.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:38.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:38.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:00:38.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:00:38.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:38.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:00:38.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:00:38.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:38.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:00:38.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:00:38.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:38.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:00:38.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:00:38.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:38.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:00:38.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:00:38.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:38.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:00:38.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:00:38.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:38.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:00:38.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:00:38.133 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:38.134 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:38.134 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:38.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.134 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:38.134 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:38.134 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 76.580476019000002, 2024-08-30T16:00:38.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:00:38.134 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:38.134 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:38.134 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:38.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:38.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:38.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:38.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:38.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:38.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:38.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:38.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:00:38.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:00:38.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:38.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:00:38.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:00:38.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:38.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:00:38.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:00:38.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:38.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:00:38.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:00:38.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:38.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:00:38.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:00:38.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:38.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:00:38.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:00:38.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:38.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:00:38.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:00:38.138 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:38.138 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:38.138 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:38.138 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:38.138 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:38.138 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:38.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:38.199 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:38.198+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:38.266 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:38.267 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:38.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:38.317 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:38.316+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:38.401 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:38.400+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:38.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:00:38.458 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:38.458 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:38.458 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:38.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:38.510 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:38.508+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:38.603 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:38.602+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:38.644 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:00:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:38.644 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 73.264385583000006, 2024-08-30T16:00:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:00:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:38.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:00:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:00:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:00:38.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:00:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:00:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:00:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:00:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:00:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:00:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:00:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:00:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:00:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:00:38.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:00:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:00:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:00:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:38.649 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:38.649 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:38.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:38.705 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:38.703+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:38.798 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:38.798 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:38.798 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:38.798 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:38.805 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:38.804+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:38.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:38.906 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:38.905+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:38.988 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:11.604 33106== Warning: unimplemented fcntl command: 1036 2024-08-30T16:00:39.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:00:39.003 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:39.004 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:39.004 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:39.006 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:39.005+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:39.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:39.056 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:11.673 33106== Warning: unimplemented fcntl command: 1036 2024-08-30T16:00:39.086 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":16,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:00:39.107 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:39.105+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:39.239 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:00:39.239 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:00:39.240 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:39.238+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:39.243 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 74.784455711999996, 2024-08-30T16:00:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:00:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:00:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:00:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:00:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:00:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:00:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:00:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:00:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:00:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:00:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:00:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:00:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:00:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:00:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:00:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:00:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:00:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:00:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:00:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:00:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:00:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:00:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:00:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:00:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:00:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:00:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:00:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:00:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:00:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:00:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 73.668589377000004, 2024-08-30T16:00:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:00:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:00:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:00:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:00:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:00:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:00:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:00:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:00:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:00:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:00:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:00:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:00:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:00:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:00:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 73.608997879, 2024-08-30T16:00:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:00:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:00:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:00:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:00:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:00:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:00:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:00:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:00:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:00:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:00:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:00:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:00:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:00:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:00:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:00:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 73.582299191999994, 2024-08-30T16:00:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:00:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:00:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:00:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:00:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:00:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:00:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:00:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:00:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:00:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:00:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:00:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:00:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 72.714682663999994, 2024-08-30T16:00:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:00:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:00:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:00:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:00:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:00:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:00:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:00:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:00:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:00:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:00:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:00:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:00:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:00:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:00:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:00:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:00:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:00:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:00:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:00:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:00:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:00:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:00:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:00:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:00:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 72.555435584999998, 2024-08-30T16:00:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:00:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:00:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:00:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:00:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:00:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:00:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:00:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:00:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:00:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:00:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:00:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:00:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:00:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:00:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:00:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 72.279261519000002, 2024-08-30T16:00:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:00:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:00:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:00:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:00:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:00:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:00:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:00:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:00:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:00:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:00:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:00:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:00:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 71.620969939999995, 2024-08-30T16:00:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:00:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:00:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:00:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:00:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:00:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:00:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:00:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:00:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:00:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:00:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:00:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:00:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:00:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:00:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:00:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:00:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:00:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:00:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:00:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:00:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:00:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:00:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:00:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:00:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 71.525056656000004, 2024-08-30T16:00:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:00:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:00:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:00:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:00:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:00:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:00:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:00:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:00:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:00:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:00:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:00:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:00:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:00:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:00:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.273696145000001, 2024-08-30T16:00:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:00:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:00:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:00:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:00:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:00:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:00:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:00:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:00:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:00:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:00:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:00:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:00:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:00:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:39.298 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:39.299 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:39.344 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:39.342+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:39.350 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:00:39.351 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:00:39.353 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:11.970 33106== Warning: unimplemented fcntl command: 1036 2024-08-30T16:00:39.409 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:39.408+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:39.424 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:39.424 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:39.424 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:39.425 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:39.433 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:39.440 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:00:39.440 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:00:39.518 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:39.516+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:39.531 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:00:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:00:39.610 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:39.609+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:39.638 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:00:39.639 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:00:39.639 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:39.639 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:39.648 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:00:39.722 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:00:39.722 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:00:39.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-08-30T16:00:39.749 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:39.747+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:39.811 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:00:39.812 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:00:39.812 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:39.810+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:39.819 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:39.819 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:39.828 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:00:39.900 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:00:39.900 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:00:39.913 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:39.911+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:39.995 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:39.995 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:40.004 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:00:40.045 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:40.043+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:40.115 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:40.113+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:40.170 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:40.171 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:40.179 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:00:40.215 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:40.214+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:40.305 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:40.305 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:40.316 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:00:40.316 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:40.315+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:40.433 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:40.432+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:40.447 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:40.447 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:40.456 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:00:40.541 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:40.539+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:40.600 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:00:40.600 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:00:40.618 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:40.617+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:40.624 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:40.624 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:40.633 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:00:40.719 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:40.717+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:40.799 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:40.799 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:40.808 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:00:40.851 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:40.849+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:40.919 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:40.918+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:40.971 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:40.971 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:40.981 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:00:41.025 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:41.024+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:41.103 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:41.103 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:41.112 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:00:41.122 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:41.120+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:41.223 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:41.221+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:41.277 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:41.277 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:41.287 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:00:41.323 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:41.322+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:41.419 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:41.419 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:41.424 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:41.422+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:41.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:00:41.525 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:41.524+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:41.622 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:41.622 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:41.626 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:41.625+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:41.629 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-08-30T16:00:41.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:00:41.728 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:41.727+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:41.819 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:41.820 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:41.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:41.829 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:41.828+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:41.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:00:41.929 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:41.928+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:41.997 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:41.998 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:42.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:42.030 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:42.029+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:42.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:00:42.146 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:42.145+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:42.156 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:00:42.156 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:00:42.157 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:42.157 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:42.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:42.231 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:42.230+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:42.259 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:00:42.259 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:00:42.332 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:42.331+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:42.344 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 81.069583387999998, 2024-08-30T16:00:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:00:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:00:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:00:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:00:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:00:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:00:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:00:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:00:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:00:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:00:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:00:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:00:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:00:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:00:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:00:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 80.796656026999997, 2024-08-30T16:00:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:00:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:00:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:00:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:00:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:00:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:00:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:00:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:00:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:00:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:00:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:00:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:00:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:00:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:00:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:00:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:42.353 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:42.353 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:00:42.353 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:00:42.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:42.433 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:42.433+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:42.450 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:00:42.450 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:00:42.507 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:42.508 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:42.508 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:42.508 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:42.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:42.549 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:00:42.549 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:00:42.549 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:42.548+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:42.632 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:00:42.632 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:00:42.635 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:42.634+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:42.645 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:42.645 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:42.645 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:42.645 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:42.645 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:42.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:42.717 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:00:42.718 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:00:42.742 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:42.742+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:42.743 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:00:42.757 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-08-30T16:00:42.757 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-30T16:00:42.757 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:00:42.773 INFO:tasks.ceph.mon.b.smithi094.stderr:2024-08-30T16:00:42.761+0000 a084640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 22550) UID: 0 2024-08-30T16:00:42.773 INFO:tasks.ceph.mon.b.smithi094.stderr:2024-08-30T16:00:42.762+0000 a084640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-08-30T16:00:42.805 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:00:42.806 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:00:42.838 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:42.838+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:42.848 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:42.848 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:42.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:42.848 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:42.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.848 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:00:42.848 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:42.848 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 77.470531797000007, 2024-08-30T16:00:42.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:00:42.849 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:42.849 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:42.849 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:42.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:42.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:42.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:42.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:42.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:42.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:42.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:42.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:00:42.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:00:42.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:42.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:00:42.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:00:42.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:42.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:00:42.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:00:42.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:42.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:00:42.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:00:42.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:42.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:00:42.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:00:42.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:42.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:00:42.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:00:42.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:42.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:00:42.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:00:42.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:42.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:42.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:42.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:00:42.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:00:42.853 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:42.853 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:42.853 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:42.853 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:42.853 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:42.853 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:42.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:42.942 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:42.941+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:43.040 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:43.040+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:43.041 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:43.041 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:43.041 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:43.041 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:43.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:43.140 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:43.139+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:43.218 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:43.218 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:43.218 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:43.218 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:43.218 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:43.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:43.268 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:43.267+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:43.342 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:43.341+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:43.417 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 78.967203029999993, 2024-08-30T16:00:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:00:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:00:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:00:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:00:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:00:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:00:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:00:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:00:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:00:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:00:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:00:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:00:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:00:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:00:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:00:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:00:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:00:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:00:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:00:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:00:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:00:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:00:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:00:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:00:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:00:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:00:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:00:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:00:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:00:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:00:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 77.851336695000001, 2024-08-30T16:00:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:00:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:00:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:00:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:00:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:00:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:00:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:00:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:00:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:00:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:00:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:00:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:00:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:00:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:00:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 77.791745196999997, 2024-08-30T16:00:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:00:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:00:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:00:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:00:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:00:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:00:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:00:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:00:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:00:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:00:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:00:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:00:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:00:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:00:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:00:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 77.765046510000005, 2024-08-30T16:00:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:00:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:00:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:00:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:00:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:00:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:00:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:00:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:00:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:00:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:00:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:00:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:00:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 76.897429982000006, 2024-08-30T16:00:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:00:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:00:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:00:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:00:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:00:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:00:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:00:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:00:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:00:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:00:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:00:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:00:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:00:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:00:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:00:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:00:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:00:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:00:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:00:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:00:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:00:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:00:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:00:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:00:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 76.738182902999995, 2024-08-30T16:00:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:00:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:00:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:00:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:00:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:00:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:00:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:00:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 76.462008836999999, 2024-08-30T16:00:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:00:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:00:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:00:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:00:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:00:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:00:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:00:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:00:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:00:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:00:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:00:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 75.803717258000006, 2024-08-30T16:00:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:00:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:00:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:43.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:43.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:00:43.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:43.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:43.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:00:43.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:00:43.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:43.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:00:43.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:00:43.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:43.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:00:43.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:00:43.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:43.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:00:43.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:00:43.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:43.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:00:43.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:00:43.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:43.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:00:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:00:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:00:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:00:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:00:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:00:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:00:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:00:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:00:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:00:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:00:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 75.707803974000001, 2024-08-30T16:00:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:00:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:00:43.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:00:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:00:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:00:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:00:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:00:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:00:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:00:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:00:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:00:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:00:43.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:00:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:00:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:43.464 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 16.456443462999999, 2024-08-30T16:00:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:00:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:43.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:00:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:00:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:00:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:00:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:00:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:00:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:00:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:00:43.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:00:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:00:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:00:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:00:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:43.469 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:43.469 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:43.470 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:43.470 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:43.469+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:43.571 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:43.571+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:43.596 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:43.596 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:43.596 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:43.596 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:43.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:43.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-08-30T16:00:43.690 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:43.689+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:43.769 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:43.769 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:43.770 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:43.770 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:43.770 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:43.773 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:43.773+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:43.779 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:00:43.875 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:43.873+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:43.943 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:43.943 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:43.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:43.944 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:00:43.944 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:43.953 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:00:43.975 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:43.974+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:44.076 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:44.076 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:44.076 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:44.076 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:44.076 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:44.075+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:44.085 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:00:44.195 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:44.194+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:44.208 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:44.208 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:44.209 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:44.209 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:44.209 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:44.218 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:00:44.279 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:44.278+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:44.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-08-30T16:00:44.380 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:44.380+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:44.435 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:44.435 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 93.570192598000006, 2024-08-30T16:00:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:00:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:00:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:00:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:00:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:00:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:00:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:00:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:00:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:00:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:00:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:00:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:00:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:00:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:00:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:00:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:00:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:00:44.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:00:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:00:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:00:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:00:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:00:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:00:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:00:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:00:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:44.441 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 92.653776578000006, 2024-08-30T16:00:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:00:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:00:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:00:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:00:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:00:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:00:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:00:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:00:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:00:44.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:00:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:00:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:00:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:00:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:00:44.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:00:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:00:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:00:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:00:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:00:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 92.652072024999995, 2024-08-30T16:00:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:00:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:00:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:00:44.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:00:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:00:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:00:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:00:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:00:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:00:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:00:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:00:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:00:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:00:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:00:44.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:00:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:00:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:00:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:00:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:00:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:00:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 92.232618217999999, 2024-08-30T16:00:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:00:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.452 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:00:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:00:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:00:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:00:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:00:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:00:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:00:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:00:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:00:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:00:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:00:44.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:00:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:00:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:00:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:00:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 92.215730043999997, 2024-08-30T16:00:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:00:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:00:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:00:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:00:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:00:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:00:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:00:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:00:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:00:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:00:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:00:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:00:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:00:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:00:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:00:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:00:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:00:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:00:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:00:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:00:44.462 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 92.202729974999997, 2024-08-30T16:00:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:00:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.463 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:00:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:00:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:00:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:00:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:00:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:00:44.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:00:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:00:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:00:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:00:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:00:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:00:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:00:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:00:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:00:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:00:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:00:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:00:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:00:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 92.200237682999997, 2024-08-30T16:00:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:00:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.468 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:00:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:00:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:00:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:00:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:00:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:00:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:00:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:00:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:00:44.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:00:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:00:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:00:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:44.473 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 92.199480234999996, 2024-08-30T16:00:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:00:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:00:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:00:44.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:00:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:00:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:00:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:00:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:00:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:00:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:00:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:00:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:00:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:00:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:00:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:00:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:00:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:00:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.478 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 92.198747784999995, 2024-08-30T16:00:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:00:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:00:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:44.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:00:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:00:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:00:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:00:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:00:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:00:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:00:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:00:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:00:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:00:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:00:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:00:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:00:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:00:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:00:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:00:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:00:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.483 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 92.196208986000002, 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:00:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:00:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:00:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:00:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:00:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:00:44.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:00:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:00:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:00:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:00:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:00:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:00:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:00:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:00:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:00:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:00:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:00:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 92.195460984999997, 2024-08-30T16:00:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:00:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:00:44.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:00:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:00:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:00:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:00:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:00:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:00:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:00:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:00:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:00:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:00:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:00:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:00:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:00:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:00:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:00:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:00:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.423344513000004, 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.494 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:00:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:00:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:00:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:00:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:00:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:00:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:00:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:00:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:00:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:00:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:00:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:00:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:00:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:00:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:00:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:00:44.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:00:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:00:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.421287367000005, 2024-08-30T16:00:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:00:44.499 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:00:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:00:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:00:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:00:44.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:00:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:00:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:00:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:00:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:00:44.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:00:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:00:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:00:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:00:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:00:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:00:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:00:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:00:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:00:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:44.504 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.158342867000002, 2024-08-30T16:00:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:00:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:00:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:00:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:00:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:00:44.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:00:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:00:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:00:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:00:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:00:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:00:44.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:00:44.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:00:44.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:00:44.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:00:44.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:00:44.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:00:44.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:00:44.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:00:44.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:00:44.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.509 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.510 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.510 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:44.510 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.157251364000004, 2024-08-30T16:00:44.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:00:44.510 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.510 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.510 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:00:44.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.510 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:44.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:44.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:00:44.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:44.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:00:44.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:00:44.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:00:44.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:00:44.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:00:44.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:00:44.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:00:44.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:00:44.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:00:44.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:00:44.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:00:44.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:00:44.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:00:44.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:00:44.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:00:44.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:00:44.514 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.515 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.515 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.515 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.515 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:44.515 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.156501778999996, 2024-08-30T16:00:44.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:00:44.515 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.515 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.515 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.515 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.515 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.515 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:00:44.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.516 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:44.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:44.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:00:44.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:44.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:00:44.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:00:44.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:00:44.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:00:44.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:00:44.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:00:44.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:00:44.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:00:44.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:00:44.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:00:44.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:00:44.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:00:44.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:00:44.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:00:44.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:00:44.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:00:44.520 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.520 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.520 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.520 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.520 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:44.520 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.122770247000005, 2024-08-30T16:00:44.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:00:44.520 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.521 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.521 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.521 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.521 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.521 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:00:44.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.521 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:44.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:44.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:00:44.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:00:44.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:00:44.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:00:44.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:00:44.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:00:44.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:00:44.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:00:44.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:00:44.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:00:44.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:00:44.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:00:44.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:00:44.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:00:44.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:00:44.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:00:44.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:00:44.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:00:44.525 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.525 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.525 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.525 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.122244238999997, 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:44.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:44.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:00:44.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:44.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:00:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:00:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:00:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:00:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:00:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:00:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:00:44.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:00:44.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:00:44.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:00:44.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:00:44.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:00:44.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:00:44.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:00:44.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:00:44.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:00:44.531 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.531 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.531 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.531 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:44.531 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:44.531 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 91.121788675999994, 2024-08-30T16:00:44.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:00:44.531 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.531 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.531 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.531 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.532 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.532 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:00:44.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.532 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:44.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:44.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:00:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:00:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:00:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:00:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:00:44.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:00:44.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:00:44.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:00:44.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:00:44.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:00:44.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:00:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:00:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:00:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:00:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:00:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:00:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:00:44.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:00:44.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.536 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.536 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:00:44.536 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:44.536 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 86.451273360000002, 2024-08-30T16:00:44.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:00:44.536 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:44.537 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:44.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:44.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:44.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:44.537 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:00:44.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.537 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:44.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:44.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:44.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:44.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:44.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:44.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:44.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:00:44.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:44.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:44.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:00:44.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:00:44.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:44.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:00:44.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:00:44.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:44.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:00:44.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:00:44.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:44.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:00:44.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:00:44.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:44.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:00:44.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:00:44.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:44.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:00:44.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:00:44.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:44.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:00:44.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:00:44.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:44.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:44.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:44.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:00:44.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:00:44.541 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.541 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.541 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.541 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:44.541 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:44.541 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:44.543 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:00:44.543 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:44.542+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:44.643 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:44.643+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:44.707 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:44.707 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:44.707 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:44.707 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:44.716 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:00:44.758 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:44.756+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:44.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:00:44.862 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:44.861+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:44.881 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:44.881 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:44.881 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:44.881 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:44.881 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:44.890 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:00:44.946 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:44.945+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:45.020 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:45.020 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:45.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:45.020 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:45.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.021 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:00:45.021 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:45.021 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 78.490854321, 2024-08-30T16:00:45.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:00:45.021 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:45.021 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:45.021 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:45.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:45.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:45.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:45.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:45.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:45.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:45.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:45.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:00:45.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:00:45.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:45.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:00:45.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:00:45.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:45.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:00:45.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:00:45.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:45.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:00:45.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:00:45.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:45.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:00:45.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:00:45.024 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:45.025 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:45.025 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:45.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.025 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:00:45.025 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:45.025 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 77.372406174999995, 2024-08-30T16:00:45.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:00:45.025 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:45.025 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:45.025 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:45.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:45.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:45.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:45.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:45.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:45.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:45.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:45.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:00:45.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:00:45.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:45.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:00:45.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:00:45.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:45.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:00:45.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:00:45.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:45.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:00:45.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:00:45.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:00:45.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:45.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:00:45.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:00:45.029 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:45.029 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:45.029 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:45.029 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:45.029 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:45.029 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:45.030 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:00:45.057 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:00:45.058 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:00:45.074 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:45.074+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:45.148 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:45.148+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:45.149 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:00:45.149 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:00:45.154 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:45.154 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:45.154 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:45.154 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:45.164 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:00:45.237 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:00:45.238 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:00:45.249 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:45.249+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:45.326 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:00:45.327 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:00:45.329 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:45.329 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:45.329 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:45.329 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:45.329 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:45.339 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:00:45.380 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:45.379+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:45.410 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:00:45.411 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:00:45.451 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:45.450+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:45.494 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:00:45.495 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:00:45.505 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:45.505 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:45.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:45.505 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:45.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.506 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:45.506 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:45.506 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 77.127493012000002, 2024-08-30T16:00:45.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:00:45.506 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:45.506 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:00:45.506 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:45.506 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:00:45.506 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:45.506 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:00:45.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.506 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:00:45.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:00:45.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:45.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:00:45.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:00:45.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:45.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:00:45.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:00:45.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:45.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:00:45.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:00:45.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:45.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:00:45.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:00:45.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:45.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:45.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:45.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:00:45.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:00:45.510 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:45.510 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:45.510 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:45.510 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:45.510 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:45.510 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:45.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:00:45.557 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:45.556+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:45.577 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:00:45.577 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:00:45.601 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:00:45.601 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:00:45.601 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:00:45.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 6 2024-08-30T16:00:45.653 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:45.652+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:45.654 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:45.654 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:45.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:00:45.668 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:00:45.669 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:00:45.754 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:45.754+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:45.800 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:45.800 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:45.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:00:45.855 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:45.854+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:45.938 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:45.938 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:45.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:45.993 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:45.991+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:46.057 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:46.055+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:46.085 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:46.085 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:46.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:46.157 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:46.157+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:46.258 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:46.258+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:46.262 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:46.262 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:46.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:46.359 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:46.358+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:46.399 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:46.399 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:46.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:46.493 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:46.492+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:46.561 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:46.560+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:46.574 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:46.575 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:46.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:46.669 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:46.668+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:46.752 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:46.752 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:46.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:46.763 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:46.761+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:46.865 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:46.863+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:46.928 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:46.928 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:46.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:46.965 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:46.963+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:47.065 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:47.065 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:47.065 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:47.064+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:47.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:47.167 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:47.165+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:47.204 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:47.204 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:47.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:47.268 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:47.266+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:47.340 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:47.340 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:47.349 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:47.390 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:47.389+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:47.475 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:47.474+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:47.489 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:47.489 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:47.499 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:47.570 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:47.569+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:47.664 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:47.664 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:47.671 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:47.669+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:47.674 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:00:47.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:00:47.771 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:47.770+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:47.838 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:47.838 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:47.847 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:00:47.891 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:47.889+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:47.918 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:00:47.919 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:00:47.970 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:47.970 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:47.974 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:47.972+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:47.980 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:00:48.012 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:00:48.013 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:00:48.076 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:48.074+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:48.103 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:00:48.103 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:00:48.144 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:48.144 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:48.153 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:00:48.176 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:48.175+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:48.195 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:00:48.195 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:00:48.277 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:48.275+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:48.282 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:00:48.282 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:00:48.370 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:00:48.370 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:00:48.441 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:48.442 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:48.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:48.442 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:48.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.442 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:48.442 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:48.442 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 97.557306629999999, 2024-08-30T16:00:48.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:00:48.442 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.442 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.443 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.443 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.443 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:00:48.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.443 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:48.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:48.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:00:48.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:00:48.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:00:48.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:00:48.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:00:48.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:00:48.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:00:48.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:00:48.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:00:48.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:00:48.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:00:48.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:00:48.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:00:48.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:00:48.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:00:48.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:00:48.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:00:48.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:00:48.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:00:48.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:00:48.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:00:48.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:00:48.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:00:48.448 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.448 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.448 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.448 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:48.448 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:48.448 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.64089061, 2024-08-30T16:00:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:00:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:00:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:48.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:48.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:00:48.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:48.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:00:48.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:00:48.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:00:48.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:00:48.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:00:48.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:00:48.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:00:48.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:00:48.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:00:48.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:00:48.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:00:48.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:00:48.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:00:48.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:00:48.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:00:48.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:00:48.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.454 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.454 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:48.455 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:48.455 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.639186057000003, 2024-08-30T16:00:48.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:00:48.455 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.455 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.455 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:00:48.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.456 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:48.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:48.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:00:48.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:00:48.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:00:48.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:00:48.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:00:48.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:00:48.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:00:48.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:00:48.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:00:48.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:00:48.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:00:48.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:00:48.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:00:48.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:00:48.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:00:48.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:00:48.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:00:48.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:00:48.463 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.463 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.463 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.464 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.464 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:48.464 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.219732250000007, 2024-08-30T16:00:48.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:00:48.464 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.464 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.464 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.464 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.465 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.465 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:00:48.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.465 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:48.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:48.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:00:48.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:00:48.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:00:48.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:00:48.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:00:48.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:00:48.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:00:48.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:00:48.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:00:48.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:00:48.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:00:48.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:00:48.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:00:48.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:00:48.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:00:48.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:00:48.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:00:48.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:00:48.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.470 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.470 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.470 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.470 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:48.470 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.202844076000005, 2024-08-30T16:00:48.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:00:48.471 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.471 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.471 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.471 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.471 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.471 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:00:48.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.471 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:48.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:48.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:00:48.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:00:48.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:00:48.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:00:48.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:00:48.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:00:48.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:00:48.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:00:48.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:00:48.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:00:48.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:00:48.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:00:48.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:00:48.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:00:48.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:00:48.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:00:48.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:00:48.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:00:48.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.476 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.476 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.476 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:48.476 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.189844007000005, 2024-08-30T16:00:48.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:00:48.476 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.476 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.476 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.477 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.477 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.477 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:00:48.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.477 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:48.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:48.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:00:48.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:00:48.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:00:48.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:00:48.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:00:48.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:00:48.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:00:48.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:00:48.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:00:48.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:00:48.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:00:48.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:00:48.481 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.481 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.482 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.482 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:48.482 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.187351715000005, 2024-08-30T16:00:48.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:00:48.482 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.482 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.482 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.482 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.483 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.483 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:00:48.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.483 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:48.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:48.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:00:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:00:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:00:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:00:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:00:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:00:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:00:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:00:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:00:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:00:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:00:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:00:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:00:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:00:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:00:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:00:48.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:00:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:48.488 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.186594267000004, 2024-08-30T16:00:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:00:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:00:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.489 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:48.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:00:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:00:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:00:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:00:48.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:00:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:00:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:00:48.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:00:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:00:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:00:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:00:48.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:00:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:00:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:00:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:00:48.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:00:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:00:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:00:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.495 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.185861817000003, 2024-08-30T16:00:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:00:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:00:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:00:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:48.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:00:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:00:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:00:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:00:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:00:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:00:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:00:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:00:48.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:00:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:00:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:00:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:00:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:00:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:00:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:00:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:00:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.502 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.183323017999996, 2024-08-30T16:00:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:00:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.503 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:00:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:00:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:00:48.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:00:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:00:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:00:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:00:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:00:48.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:00:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:00:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:00:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:00:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:00:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:00:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:00:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:00:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:00:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:00:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.508 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 96.182575017000005, 2024-08-30T16:00:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:00:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.509 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:00:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:00:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:00:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:00:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:00:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:00:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:00:48.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:00:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:00:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:00:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:00:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:00:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:00:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:00:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:00:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:00:48.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:00:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:00:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:00:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.410458544999997, 2024-08-30T16:00:48.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:00:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:00:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:00:48.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:00:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:00:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:00:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:00:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:00:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:00:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:00:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:00:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:00:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:00:48.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:00:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:00:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:00:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:00:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:00:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.408401398999999, 2024-08-30T16:00:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:00:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:00:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.520 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:48.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:00:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:00:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:00:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:00:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:00:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:00:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:00:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:00:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:00:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:00:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:00:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:00:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:00:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:00:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:00:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:00:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:00:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.145456898999996, 2024-08-30T16:00:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:00:48.526 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:00:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:00:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:00:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:00:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:00:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:00:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:00:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:00:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:00:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:00:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:00:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:00:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:00:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:00:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:00:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:00:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:00:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:00:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:00:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.144365395999998, 2024-08-30T16:00:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:00:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:00:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:00:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:00:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:00:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:00:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:00:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:00:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:00:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:00:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:00:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:00:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:00:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:00:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:00:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:00:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:00:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:00:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:00:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.143615811000004, 2024-08-30T16:00:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:00:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:00:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:00:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:00:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:00:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:00:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:00:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:00:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:00:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:00:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:00:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:00:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:00:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:00:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:00:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:00:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:00:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:00:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:00:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.109884278999999, 2024-08-30T16:00:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:00:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:00:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:00:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:00:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:00:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:00:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:00:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:00:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:00:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:00:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:00:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:00:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:00:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:00:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:00:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:00:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:00:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:00:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:00:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:00:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.109358271000005, 2024-08-30T16:00:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:00:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:00:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:00:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:00:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:00:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:00:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:00:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:00:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:00:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:00:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:00:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:00:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:00:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:00:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:00:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:00:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:00:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:00:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:00:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 95.108902708000002, 2024-08-30T16:00:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:00:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:00:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:00:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:00:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:00:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:00:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:00:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:00:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:00:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:00:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:00:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:00:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:00:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:00:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:00:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:00:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:00:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:00:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:00:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:00:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:00:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 90.438387391999996, 2024-08-30T16:00:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:00:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:00:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:00:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:00:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:00:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:00:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:00:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:00:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:00:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:00:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:00:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:00:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:00:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:00:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:00:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:00:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:00:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:00:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:00:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:48.571 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:48.571 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:48.439+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:48.572 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:00:48.573 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:00:48.574 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:00:48.574 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:00:48.574 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:00:48.575 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:48.573+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:48.675 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:48.674+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:48.698 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:48.699 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:48.699 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:48.699 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:48.708 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:00:48.776 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:48.774+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:48.834 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:48.834 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:48.835 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:48.835 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:48.843 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:00:48.860 INFO:tasks.ceph.mon.b:Stopped 2024-08-30T16:00:48.860 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2024-08-30T16:00:48.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:00:48.904 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:48.903+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:48.977 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:48.976+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:49.011 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:49.011 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:49.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:49.011 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:49.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.011 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:00:49.011 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:49.011 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 82.481486892000007, 2024-08-30T16:00:49.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:00:49.012 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:49.012 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:49.012 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:49.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:49.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:49.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:49.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:49.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:49.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:49.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:49.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:00:49.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:00:49.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:49.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:00:49.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:00:49.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:00:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:00:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:00:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:00:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:00:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:00:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:49.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:00:49.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:00:49.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:49.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:00:49.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:00:49.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:49.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:00:49.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:00:49.016 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:49.016 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:49.016 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:49.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.016 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:00:49.016 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:49.016 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 81.363038746000001, 2024-08-30T16:00:49.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:00:49.016 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:49.016 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:49.016 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:49.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:49.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:49.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:49.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:49.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:49.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:49.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:49.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:00:49.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:00:49.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:49.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:00:49.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:00:49.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:49.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:00:49.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:00:49.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:49.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:00:49.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:00:49.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:49.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:00:49.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:00:49.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:49.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:00:49.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:00:49.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:49.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:00:49.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:00:49.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:49.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:00:49.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:00:49.020 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:49.020 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:49.020 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:49.021 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:49.021 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:49.021 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:49.022 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:00:49.078 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:49.077+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:49.180 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:49.178+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:49.187 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:49.187 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:49.187 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:49.196 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:00:49.280 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:49.279+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:49.360 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:49.360 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:49.360 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:49.360 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:49.360 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:49.370 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:00:49.381 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:49.380+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:49.482 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:49.481+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:49.495 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 81.117289041000006, 2024-08-30T16:00:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:00:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:00:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:00:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:00:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:00:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:00:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:00:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:00:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:00:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:00:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:00:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:00:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:00:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:00:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:00:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:00:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:49.499 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:49.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:00:49.584 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:49.582+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:49.638 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:49.638 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:49.638 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:49.638 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:49.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:00:49.685 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:49.683+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:49.774 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:49.774 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:49.774 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:49.774 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:49.774 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:49.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:00:49.786 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:49.784+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:49.887 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:49.886+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:49.909 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:49.910 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:49.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:49.910 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:00:49.910 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:49.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:50.003 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:50.002+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:50.087 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:50.087 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:50.087 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:50.088 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:50.089 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:50.087+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:50.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:50.190 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:50.189+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:50.264 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:50.264 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:50.264 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:50.264 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:50.265 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:50.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:50.317 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:50.316+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:50.392 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:50.391+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:50.406 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 89.131481540999999, 2024-08-30T16:00:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:00:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:50.406 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:00:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:00:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:00:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:00:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:50.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:00:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:00:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:00:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:00:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:00:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:00:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:00:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:00:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:00:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:00:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 88.858554179999999, 2024-08-30T16:00:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:00:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:50.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:00:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:00:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:00:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:00:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:00:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:00:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:00:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:00:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:00:50.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:00:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:00:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:00:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:00:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:00:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:50.415 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:50.416 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:50.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:50.501 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:50.500+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:50.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-08-30T16:00:50.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:00:50.585 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:50.585 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:50.585 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:50.585 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:50.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:50.595 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:50.593+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:50.695 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:50.694+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:50.763 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:50.764 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:50.764 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:50.764 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:50.764 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:50.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:50.815 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:50.813+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:50.898 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:50.896+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:50.942 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:50.942 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:50.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:50.942 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:50.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.943 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:00:50.943 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:50.943 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 85.564894203999998, 2024-08-30T16:00:50.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:00:50.943 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:50.943 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:50.943 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:50.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:50.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:50.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:50.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:50.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:50.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:50.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:50.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:00:50.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:00:50.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:50.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:00:50.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:00:50.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:50.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:00:50.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:00:50.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:50.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:00:50.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:00:50.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:50.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:00:50.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:00:50.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:50.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:00:50.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:00:50.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:50.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:00:50.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:00:50.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:50.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:50.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:50.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:00:50.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:00:50.947 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:50.947 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:50.947 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:50.947 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:50.947 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:50.947 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:50.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:50.999 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:50.997+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:51.106 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:51.105+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:51.120 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:51.120 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:51.120 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:51.120 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:51.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:51.200 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:51.199+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:51.299 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:51.299 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:51.300 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:51.300 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:51.300 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:51.301 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:51.300+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:51.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:51.402 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:51.401+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:51.458 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:51.458 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 87.007910219999999, 2024-08-30T16:00:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:00:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:51.459 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:00:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:00:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:00:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:00:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:00:51.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:00:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:00:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:00:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:00:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:00:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:00:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:00:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:00:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:00:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:00:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:00:51.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:00:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:00:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:00:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:00:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:00:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:00:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:00:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:00:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:00:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:00:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:51.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:00:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:00:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:00:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 85.892043885000007, 2024-08-30T16:00:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:00:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:51.466 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:51.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:00:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:00:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:00:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:00:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:00:51.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:00:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:00:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:00:51.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:00:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:00:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:00:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:00:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:00:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.470 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 85.832452387000004, 2024-08-30T16:00:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:00:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:51.471 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:00:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:00:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:00:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:00:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:00:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:51.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:00:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:00:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:00:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:00:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:00:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:00:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:00:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:00:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:00:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 85.805753699999997, 2024-08-30T16:00:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:00:51.475 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:00:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:00:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:00:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:00:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:00:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:00:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:00:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:00:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:00:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:00:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:00:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 84.938137171999998, 2024-08-30T16:00:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:00:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:00:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:00:51.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:00:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:00:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:00:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:00:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:00:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:00:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:00:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:00:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:00:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:00:51.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:00:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:00:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:00:51.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:00:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:00:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:00:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:00:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:00:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:00:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:00:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:00:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 84.778890093000001, 2024-08-30T16:00:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:00:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:51.486 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:51.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:00:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:00:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:00:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:00:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:00:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:00:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:51.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:00:51.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:00:51.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:51.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:00:51.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:00:51.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:51.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:00:51.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:00:51.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:51.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:00:51.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:00:51.490 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.490 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:51.490 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.490 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:51.490 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:51.490 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 84.502716027000005, 2024-08-30T16:00:51.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:00:51.491 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:51.491 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:51.491 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:51.491 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:51.491 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:51.491 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:51.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.491 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:51.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:51.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:51.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:51.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:51.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:51.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:00:51.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:51.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:51.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:00:51.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:00:51.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:51.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:00:51.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:00:51.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:51.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:00:51.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:00:51.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:51.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:00:51.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:00:51.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:51.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:00:51.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:00:51.494 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.494 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:51.494 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 83.844424447999998, 2024-08-30T16:00:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:00:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:00:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:51.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:51.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:51.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:51.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:51.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:51.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:00:51.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:51.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:51.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:00:51.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:00:51.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:51.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:00:51.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:00:51.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:51.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:00:51.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:00:51.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:51.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:00:51.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:00:51.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:51.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:00:51.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:00:51.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:00:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:00:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:00:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:00:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:00:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:00:51.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:51.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:00:51.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:00:51.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:51.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:00:51.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:00:51.500 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.500 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:51.500 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.501 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:00:51.501 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:51.501 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 83.748511164000007, 2024-08-30T16:00:51.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:00:51.501 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:51.501 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:51.501 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:51.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:51.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:51.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:51.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:51.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:51.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:00:51.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:51.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:00:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:00:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:00:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:00:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:00:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:00:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:00:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:00:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.497150652999998, 2024-08-30T16:00:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:00:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:00:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:00:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:00:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:00:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:00:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:00:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:00:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:00:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:00:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:00:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:00:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:00:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:51.509 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:51.511 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:51.511 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:51.509+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:51.612 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:51.610+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:51.651 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:51.651 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:51.651 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:51.651 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:51.661 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:51.713 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:51.711+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:51.813 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:51.811+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:51.825 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:51.825 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:51.826 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:51.826 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:51.835 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:00:51.920 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:51.918+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:51.999 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:52.000 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:52.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:52.000 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:00:52.000 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:52.010 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:00:52.053 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:52.050+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:52.116 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:52.115+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:52.173 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:52.174 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:52.174 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:52.174 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:52.183 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:00:52.228 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:52.226+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:52.308 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:52.308 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:52.308 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:52.308 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:52.309 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:52.317 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:00:52.318 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:52.315+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:52.419 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:52.417+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:52.519 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:52.518+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:52.529 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:52.529 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 101.667606574, 2024-08-30T16:00:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:00:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:00:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:00:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:00:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:00:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:00:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:00:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:00:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:00:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:00:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:00:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:00:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:00:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:00:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:00:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:00:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:00:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:00:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:00:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:00:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:00:52.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:00:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:00:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:00:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:00:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 100.751190554, 2024-08-30T16:00:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:00:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:00:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:52.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:00:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:00:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:00:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:00:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:00:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:00:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:00:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:00:52.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:00:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:00:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:00:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:00:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:00:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:00:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:00:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:00:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:00:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 100.74948600099999, 2024-08-30T16:00:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:00:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:00:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:00:52.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:00:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:00:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:00:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:00:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:00:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:00:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:00:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:00:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:00:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:00:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:00:52.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:00:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:00:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:00:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:00:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:00:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:00:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 100.330032194, 2024-08-30T16:00:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:00:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.548 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:00:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:00:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:00:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:00:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:00:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:00:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:00:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:00:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:00:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:00:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:00:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:00:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:00:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:00:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:00:52.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:00:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:00:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:00:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:00:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 100.31314402, 2024-08-30T16:00:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:00:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:00:52.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:00:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:00:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:00:52.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:00:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:00:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:00:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:00:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:00:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:00:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:00:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:00:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:00:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:00:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:00:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:00:52.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:00:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:00:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:00:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 100.300143951, 2024-08-30T16:00:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:00:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:00:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:00:52.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:00:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:00:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:00:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:00:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:00:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:00:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:00:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:00:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:00:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:00:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:00:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:00:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:00:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:00:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:00:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:00:52.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:00:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 100.297651659, 2024-08-30T16:00:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:00:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.566 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:00:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:00:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:00:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:00:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:00:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:00:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:00:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:00:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:00:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:00:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:00:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:00:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:00:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:00:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:00:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:00:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:00:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:00:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.571 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 100.29689421099999, 2024-08-30T16:00:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:00:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:00:52.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:00:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:00:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:00:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:00:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:00:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:00:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:00:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:00:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:00:52.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:00:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:00:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:00:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:00:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:00:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:00:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:00:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:00:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:00:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 100.29616176099999, 2024-08-30T16:00:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:00:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:00:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:00:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:52.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:00:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:00:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:00:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:00:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:00:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:00:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:00:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:00:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:00:52.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:00:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:00:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:00:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:00:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:00:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:00:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:00:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 100.293622962, 2024-08-30T16:00:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:00:52.583 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:00:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.584 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:00:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:00:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:00:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:00:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:00:52.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:00:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:00:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:00:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:00:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:00:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:00:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:00:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:00:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:00:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:00:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:00:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:00:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 100.292874961, 2024-08-30T16:00:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:00:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:00:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.590 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:00:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:00:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:00:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:00:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:00:52.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:00:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:00:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:00:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:00:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:00:52.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:00:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:00:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:00:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:00:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:00:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:00:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:00:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:00:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.520758489000002, 2024-08-30T16:00:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:00:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:00:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:00:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:00:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:00:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:00:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:00:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:00:52.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:00:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:00:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:00:52.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:00:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:00:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:00:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:00:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:00:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:00:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:00:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:00:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:00:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.601 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.518701343000004, 2024-08-30T16:00:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:00:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:00:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:00:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:52.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:00:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:00:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:00:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:00:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:00:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:00:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:00:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:00:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:00:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:00:52.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:00:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:00:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:00:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:00:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:00:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:00:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.255756843, 2024-08-30T16:00:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:00:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.607 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:00:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:00:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:00:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:00:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:00:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:00:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:00:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:00:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:00:52.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:00:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:00:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:00:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:00:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:00:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:00:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:00:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:00:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:00:52.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:00:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.254665340000003, 2024-08-30T16:00:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:00:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.613 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:00:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:52.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:00:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:00:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:00:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:00:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:00:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:00:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:00:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:00:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:00:52.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:00:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:00:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:00:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:00:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:00:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:00:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:00:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:00:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.618 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.253915754999994, 2024-08-30T16:00:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:00:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:00:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:00:52.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:00:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:00:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:00:52.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:00:52.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:00:52.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:00:52.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:00:52.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:00:52.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:00:52.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:00:52.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:00:52.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:00:52.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:00:52.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:00:52.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:00:52.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:00:52.625 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.625 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.625 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.626 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.626 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:52.626 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.220184223000004, 2024-08-30T16:00:52.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:00:52.626 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.627 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.627 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.627 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.627 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.627 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:00:52.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.627 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:52.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:52.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:00:52.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:00:52.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:00:52.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:00:52.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:00:52.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:00:52.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:00:52.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:00:52.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:00:52.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:00:52.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:00:52.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:00:52.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:00:52.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:00:52.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:00:52.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:00:52.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:00:52.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:00:52.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.637 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.637 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.638 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:52.638 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.219658214999995, 2024-08-30T16:00:52.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:00:52.638 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.638 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.639 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.639 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.639 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.639 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:00:52.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.639 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:52.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:52.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:00:52.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:52.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:00:52.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:00:52.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:00:52.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:00:52.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:00:52.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:00:52.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:00:52.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:00:52.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:00:52.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:00:52.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:00:52.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:00:52.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:00:52.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:00:52.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:00:52.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:00:52.645 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.645 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.645 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.645 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:52.645 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:52.645 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 99.219202652000007, 2024-08-30T16:00:52.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:00:52.645 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.646 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.646 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.646 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.646 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.646 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:00:52.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.646 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:52.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:52.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:00:52.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:00:52.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:00:52.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:00:52.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:00:52.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:00:52.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:00:52.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:00:52.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:00:52.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:00:52.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:00:52.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:00:52.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:00:52.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:00:52.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:00:52.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:00:52.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:00:52.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:00:52.651 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.651 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.652 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.652 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:00:52.652 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:52.652 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.548687336, 2024-08-30T16:00:52.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:00:52.652 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:52.652 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:52.652 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:52.652 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:52.652 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:00:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:52.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:00:52.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:52.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:52.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:00:52.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:00:52.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:52.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:00:52.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:00:52.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:52.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:00:52.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:00:52.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:52.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:00:52.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:00:52.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:52.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:00:52.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:00:52.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:52.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:00:52.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:00:52.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:52.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:00:52.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:00:52.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:52.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:52.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:52.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:00:52.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:00:52.657 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.657 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.657 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.657 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:52.657 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:52.657 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:52.659 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:00:52.659 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:52.657+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:52.760 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:52.759+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:52.827 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:52.827 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:52.827 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:52.827 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:52.837 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:00:52.861 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:52.859+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:52.961 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:52.961 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:52.961 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:52.961 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:52.961 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:52.961 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:52.960+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:52.970 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:00:53.080 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:53.078+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:53.097 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:00:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 86.567347123000005, 2024-08-30T16:00:53.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:00:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:53.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:00:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:00:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:00:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:00:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:00:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:00:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:00:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:00:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:00:53.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:00:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:00:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:00:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:00:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:00:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:53.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:00:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:00:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:00:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 85.448898976999999, 2024-08-30T16:00:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:00:53.102 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:53.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:00:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:00:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:00:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:00:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:00:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:00:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:00:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:00:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:53.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:00:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:00:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:00:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:00:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:00:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:00:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:00:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:00:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:53.107 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:53.107 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:53.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:00:53.193 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:53.191+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:53.264 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:53.263+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:53.272 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:53.272 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:53.272 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:53.272 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:53.281 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:00:53.366 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:53.364+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:53.446 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:53.446 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:53.446 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:53.456 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:00:53.468 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:53.465+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:53.568 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:53.566+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:53.621 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:53.621 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:53.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:53.621 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:53.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.622 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:53.622 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:53.622 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 85.243422269999996, 2024-08-30T16:00:53.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:00:53.622 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:53.622 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:00:53.622 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:53.622 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:00:53.622 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:53.622 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:00:53.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.622 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:53.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:53.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:53.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:53.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:53.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:53.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:53.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:53.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:00:53.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:00:53.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:53.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:00:53.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:00:53.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:53.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:00:53.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:00:53.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:53.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:00:53.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:00:53.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:53.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:00:53.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:00:53.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:53.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:53.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:53.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:00:53.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:00:53.626 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:53.626 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:53.626 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:53.626 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:53.627 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:53.627 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:53.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:00:53.668 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:53.667+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:53.769 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:53.767+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:53.771 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:53.771 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:53.771 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:53.771 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:53.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:00:53.870 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:53.869+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:53.908 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:53.908 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:53.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:00:54.002 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:54.001+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:54.072 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:54.071+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:54.085 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:54.086 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:54.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:54.086 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:00:54.086 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:54.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:54.173 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:54.172+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:54.263 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:54.263 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:54.264 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:54.265 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:54.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:54.275 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:54.273+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:54.390 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:54.389+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:54.401 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:54.401 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:54.401 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:54.401 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:54.402 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:54.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:54.476 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:54.474+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:54.577 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:54.576+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:54.584 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:54.584 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 93.310152736000006, 2024-08-30T16:00:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:00:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:54.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:00:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:00:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:00:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:00:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:00:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:00:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:00:54.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:00:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:00:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:00:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:00:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:00:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:54.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:00:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:00:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 93.037225375000006, 2024-08-30T16:00:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:00:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:54.589 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:00:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:00:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:00:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:00:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:54.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:00:54.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:00:54.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:54.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:00:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:00:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:00:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:00:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:00:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:00:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:54.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:54.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:54.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:00:54.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:00:54.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:54.594 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:54.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:54.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:54.594 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:54.594 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:54.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:54.680 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:54.679+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:54.764 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:54.765 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:54.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:54.779 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:54.778+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:54.880 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:54.879+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:54.942 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:54.942 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:54.942 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:54.942 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:54.942 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:54.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:54.993 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:54.991+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:55.081 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:55.079+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:55.121 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:55.121 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:55.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:55.121 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:55.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.121 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:00:55.121 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:55.121 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 89.743682746999994, 2024-08-30T16:00:55.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:00:55.122 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:55.122 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:55.122 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:55.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.122 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:55.122 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:55.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.122 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:55.122 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:00:55.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:00:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:00:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:00:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:00:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:00:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:00:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:00:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:00:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:00:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:00:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:55.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:00:55.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:00:55.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:55.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:00:55.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:00:55.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:55.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:00:55.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:00:55.126 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.126 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:55.126 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.126 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.126 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:55.126 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:55.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:55.182 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:55.180+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:55.260 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:55.260 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:55.260 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:55.260 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:55.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:55.282 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:55.281+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:55.383 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:55.382+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:55.396 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:55.397 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:55.397 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:55.397 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:55.397 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:55.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:55.484 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:55.483+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:55.586 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:55.584+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:55.597 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:55.597 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:55.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:55.597 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:55.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.597 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:55.597 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:55.597 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 91.146758843000001, 2024-08-30T16:00:55.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:00:55.598 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:55.598 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:55.598 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:55.598 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:55.598 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:55.598 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:55.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.598 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:55.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:55.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:55.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:55.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:00:55.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:55.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:00:55.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:00:55.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:55.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:00:55.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:00:55.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:55.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:00:55.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:00:55.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:55.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:00:55.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:00:55.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:55.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:00:55.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:00:55.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:00:55.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:00:55.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:00:55.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:00:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:00:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:00:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:00:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:00:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:00:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:00:55.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:55.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:00:55.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:00:55.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:55.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:00:55.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:00:55.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:55.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:00:55.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:00:55.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:55.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:00:55.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:00:55.604 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.604 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:55.604 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.604 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:00:55.604 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:55.605 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 90.030892507999994, 2024-08-30T16:00:55.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:00:55.605 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:55.605 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:55.605 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:55.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:55.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:55.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:55.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:00:55.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:55.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:00:55.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:55.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:55.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:00:55.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:00:55.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:55.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:00:55.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:00:55.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:55.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:00:55.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:00:55.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:55.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:00:55.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:00:55.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:55.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:00:55.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:00:55.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:55.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:00:55.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:00:55.609 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.609 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:55.609 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.609 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:00:55.609 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:55.609 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 89.971301010000005, 2024-08-30T16:00:55.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:00:55.609 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:55.609 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:55.609 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:55.609 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:55.610 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:55.610 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:00:55.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.610 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:55.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:55.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:55.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:55.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:00:55.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:55.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:00:55.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:00:55.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:55.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:00:55.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:00:55.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:55.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:00:55.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:00:55.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:55.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:00:55.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:00:55.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:55.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:00:55.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:00:55.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:00:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:00:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:00:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 89.944602322999998, 2024-08-30T16:00:55.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:55.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:00:55.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:55.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:00:55.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:55.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:55.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:00:55.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:00:55.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:55.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:00:55.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:00:55.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:55.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:00:55.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:00:55.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:55.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:00:55.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:00:55.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:55.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:00:55.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:00:55.617 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.617 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:55.617 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.617 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:55.617 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:55.618 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 89.076985794999999, 2024-08-30T16:00:55.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:00:55.618 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:55.618 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:55.618 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:55.618 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:55.618 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:55.618 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:00:55.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.618 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:55.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:55.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:00:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:00:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:00:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:00:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:00:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:00:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:00:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:00:55.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:55.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:00:55.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:00:55.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:55.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:00:55.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:00:55.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:55.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:00:55.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:00:55.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:55.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:00:55.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:00:55.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:55.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:00:55.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:00:55.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:55.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:00:55.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:00:55.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:55.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:00:55.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:00:55.623 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.623 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:55.623 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.624 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:00:55.624 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:55.624 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 88.917738716000002, 2024-08-30T16:00:55.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:00:55.624 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:55.624 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:55.624 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:55.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:55.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:55.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:00:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:00:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:00:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:00:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:00:55.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:55.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:00:55.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:00:55.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:55.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:00:55.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:00:55.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:55.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:00:55.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:00:55.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:55.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:00:55.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:00:55.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:55.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:00:55.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:00:55.627 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.627 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:55.628 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.628 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:55.628 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:55.628 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 88.641564650000007, 2024-08-30T16:00:55.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:00:55.628 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:55.628 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:55.628 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:55.628 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:55.628 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:55.628 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:00:55.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.629 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:55.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:55.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:55.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:55.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:00:55.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:55.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:00:55.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:00:55.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:55.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:00:55.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:00:55.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:55.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:00:55.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:00:55.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:55.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:00:55.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:00:55.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:55.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:00:55.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:00:55.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:55.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:00:55.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:00:55.632 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.632 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:55.632 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.632 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:55.632 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:55.632 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 87.983273070999999, 2024-08-30T16:00:55.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:00:55.633 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:55.633 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:55.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:55.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:00:55.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:00:55.633 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:00:55.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.633 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:55.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:55.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:55.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:00:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:00:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:00:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:00:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:00:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:00:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:00:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:00:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:55.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:00:55.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:00:55.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:55.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:00:55.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:00:55.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:00:55.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:00:55.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:00:55.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:00:55.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:00:55.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:00:55.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:00:55.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:00:55.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:00:55.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:00:55.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:00:55.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:00:55.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:55.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:00:55.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:00:55.638 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.638 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:55.638 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.638 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:00:55.638 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:55.639 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 87.887359786999994, 2024-08-30T16:00:55.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:00:55.639 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:55.639 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:55.639 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:55.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:55.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:55.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:55.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:00:55.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:55.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:00:55.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:55.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:55.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:00:55.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:00:55.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:55.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:00:55.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:00:55.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:55.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:00:55.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:00:55.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:55.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:00:55.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:00:55.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:55.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:00:55.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:00:55.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:55.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:00:55.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:00:55.642 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.642 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:55.642 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.643 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:00:55.643 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:55.643 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 28.635999276, 2024-08-30T16:00:55.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:00:55.643 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:55.643 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:00:55.643 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:00:55.643 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:00:55.643 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:55.643 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:00:55.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.643 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:55.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:55.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:55.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:55.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:00:55.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:55.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:55.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:00:55.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:00:55.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:55.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:00:55.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:00:55.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:55.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:00:55.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:00:55.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:55.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:00:55.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:00:55.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:55.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:00:55.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:00:55.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:55.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:55.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:55.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:00:55.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:00:55.646 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.647 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:55.647 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.647 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:55.647 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:55.647 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:55.648 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:55.687 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:55.686+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:55.789 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:55.787+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:55.792 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:55.793 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:55.803 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:55.889 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:55.888+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:55.974 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:55.974 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:55.974 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:55.974 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:55.974 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:55.984 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:00:55.990 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:55.989+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:56.090 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:56.089+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:56.151 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:56.152 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:56.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:56.152 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:00:56.152 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:56.161 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:00:56.191 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:56.190+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:56.292 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:56.291+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:56.326 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:56.326 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:56.326 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:56.336 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:00:56.394 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:56.393+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:56.458 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:56.458 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:56.458 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:56.458 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:56.458 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:56.466 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:00:56.494 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:56.493+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:56.595 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:56.594+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:56.641 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:56.641 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:56.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:56.642 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:56.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.642 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:56.642 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:56.642 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 105.776284054, 2024-08-30T16:00:56.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:00:56.642 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.642 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.642 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.642 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.642 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.642 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:00:56.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.643 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:56.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:00:56.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:00:56.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:00:56.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:00:56.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:00:56.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:00:56.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:00:56.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:00:56.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:00:56.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:00:56.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:00:56.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:00:56.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:00:56.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:00:56.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:00:56.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:00:56.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:00:56.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:00:56.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:00:56.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:00:56.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:00:56.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:00:56.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:00:56.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:00:56.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.648 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.648 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.648 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:56.648 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:56.648 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.859868034, 2024-08-30T16:00:56.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:00:56.648 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.648 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.648 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.649 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.649 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.649 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:00:56.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.649 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:56.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:00:56.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:00:56.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:56.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:00:56.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:00:56.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:00:56.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:00:56.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:00:56.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:00:56.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:00:56.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:00:56.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:00:56.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:00:56.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:00:56.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:00:56.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:00:56.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:00:56.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:00:56.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:00:56.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.653 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.653 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:00:56.654 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:56.654 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.85816348100001, 2024-08-30T16:00:56.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:00:56.654 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.654 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.654 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:00:56.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.654 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:56.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:00:56.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:00:56.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:00:56.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:00:56.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:00:56.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:00:56.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:00:56.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:00:56.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:00:56.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:00:56.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:00:56.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:00:56.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:00:56.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:00:56.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:00:56.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:00:56.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:00:56.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:00:56.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:00:56.658 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.659 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.659 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.659 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:56.659 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.43870967399999, 2024-08-30T16:00:56.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:00:56.659 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.659 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.659 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.659 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.659 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.660 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:00:56.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.660 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:56.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:00:56.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:00:56.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:00:56.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:00:56.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:00:56.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:00:56.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:00:56.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:00:56.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:00:56.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:00:56.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:00:56.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:00:56.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:00:56.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:00:56.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:00:56.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:00:56.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:00:56.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:00:56.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:00:56.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.664 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.664 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.664 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.42182149999999, 2024-08-30T16:00:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:00:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:00:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:56.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:00:56.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:00:56.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:00:56.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:00:56.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:00:56.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:00:56.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:00:56.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:00:56.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:00:56.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:00:56.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:00:56.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:00:56.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:00:56.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:00:56.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:00:56.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:00:56.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:00:56.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:00:56.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:00:56.669 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.670 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.670 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.670 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:56.670 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.40882143100001, 2024-08-30T16:00:56.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:00:56.670 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.670 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.671 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.671 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:00:56.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.671 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:56.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:00:56.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:00:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:00:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:00:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:00:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:00:56.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:00:56.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:00:56.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:00:56.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:00:56.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:00:56.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:00:56.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:00:56.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:00:56.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:00:56.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:00:56.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:00:56.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:00:56.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:00:56.675 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.675 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.675 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.675 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.675 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:56.676 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.40632913899999, 2024-08-30T16:00:56.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:00:56.676 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.676 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.676 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.676 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.676 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.676 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:00:56.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.676 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:56.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:00:56.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:00:56.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:00:56.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:00:56.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:00:56.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:00:56.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:00:56.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:00:56.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:00:56.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:00:56.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:00:56.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:00:56.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:00:56.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:00:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:00:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:00:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:00:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:00:56.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:00:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.40557169100001, 2024-08-30T16:00:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:00:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:00:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:00:56.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:00:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:00:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:00:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:00:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:00:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:00:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:00:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:00:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:00:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:00:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:00:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:00:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:00:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:00:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:00:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:00:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:00:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:00:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.404839241, 2024-08-30T16:00:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:00:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:00:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.687 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:00:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:00:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:00:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:00:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:00:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:00:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:00:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:00:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:00:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:00:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:00:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:00:56.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:00:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:00:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:00:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:00:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:00:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:00:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:00:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.402300442, 2024-08-30T16:00:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:00:56.692 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:00:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:00:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:00:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:00:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:00:56.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:00:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:00:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:00:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:00:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:00:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:00:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:00:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:00:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:00:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:00:56.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:00:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:00:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:00:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:00:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 104.40155244100001, 2024-08-30T16:00:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:00:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:00:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:00:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:00:56.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:00:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:00:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:00:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:00:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:00:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:00:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:00:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:00:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:00:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:00:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:00:56.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:00:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:00:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:00:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:00:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:00:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:00:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.629435969, 2024-08-30T16:00:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:00:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:00:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:56.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:00:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:00:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:00:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:00:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:00:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:00:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:00:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:00:56.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:00:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:00:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:00:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:00:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:00:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:00:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:00:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:00:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:00:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:00:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:00:56.708 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.627378823, 2024-08-30T16:00:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:00:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.709 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:00:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:00:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:00:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:00:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:00:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:00:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:00:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:00:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:00:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:00:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:00:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:00:56.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:00:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:00:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:00:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:00:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:00:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:00:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:00:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:00:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:56.714 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.364434323, 2024-08-30T16:00:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:00:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:00:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:56.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:00:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:00:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:00:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:00:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:00:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:00:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:00:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:00:56.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:00:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:00:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:00:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:00:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:00:56.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:00:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:00:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:00:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:00:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:00:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:00:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.36334282, 2024-08-30T16:00:56.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:00:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:00:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:56.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:00:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:00:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:00:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:00:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:00:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:00:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:00:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:00:56.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:00:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:00:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:00:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:00:56.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:00:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:00:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:00:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:00:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:00:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:00:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:00:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.36259323500001, 2024-08-30T16:00:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:00:56.726 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:00:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:00:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:00:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:00:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:00:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:00:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:00:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:00:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:00:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:00:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:00:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:00:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:00:56.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:00:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:00:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:00:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:00:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:00:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:00:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:00:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.328861703, 2024-08-30T16:00:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:00:56.732 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:00:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:00:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:00:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:00:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:00:56.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:00:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:00:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:00:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:00:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:00:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:00:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:00:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:00:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:00:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:00:56.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:00:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:00:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:00:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:00:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:00:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.737 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.32833569500001, 2024-08-30T16:00:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:00:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:00:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:00:56.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:00:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:00:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:00:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:00:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:00:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:00:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:00:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:00:56.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:00:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:00:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:00:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:00:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:00:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:00:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:00:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:00:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:00:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.743 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:00:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.327880132, 2024-08-30T16:00:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:00:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:00:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:00:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.745 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:00:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:00:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:00:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:00:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:00:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:00:56.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:00:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:00:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:00:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:00:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:00:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:00:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:00:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:00:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:00:56.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:00:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:00:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:00:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:00:56.749 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 98.657364815999998, 2024-08-30T16:00:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:00:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:00:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:00:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:00:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:00:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:00:56.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:00:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:00:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:00:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:00:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:00:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:00:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:00:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:00:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:00:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:00:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:00:56.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:00:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:00:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:00:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:00:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:00:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:00:56.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:00:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:56.755 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:56.755 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:56.756 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:00:56.757 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:56.755+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:56.857 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:56.856+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:56.919 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:56.919 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:56.920 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:56.920 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:56.929 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:00:56.958 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:56.957+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:57.052 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:57.052 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:57.052 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:57.052 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:57.052 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:57.059 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:57.057+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:57.062 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:00:57.160 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:57.159+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:57.228 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:00:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 90.699350167000006, 2024-08-30T16:00:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:00:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:00:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:00:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:00:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:00:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:00:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:00:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:00:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:00:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:00:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:00:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:00:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:00:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:00:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:00:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:00:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:00:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:00:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:00:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.580902021, 2024-08-30T16:00:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:00:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:00:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:00:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:00:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:00:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:00:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:00:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:00:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:00:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:00:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:00:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:00:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:00:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:00:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:00:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:00:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:00:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:00:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:00:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:00:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:00:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:57.238 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:57.239 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:00:57.284 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:57.282+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:57.363 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:57.360+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:57.365 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:57.365 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:57.365 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:00:57.365 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:57.374 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:00:57.463 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:57.462+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:57.537 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:57.537 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:00:57.537 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:00:57.537 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:00:57.538 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:57.546 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:00:57.591 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:57.589+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:57.665 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:57.664+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:57.673 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:00:57.673 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:00:57.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:00:57.673 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:00:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:00:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 89.295642732999994, 2024-08-30T16:00:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:00:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:00:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:00:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:00:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:00:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:00:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:00:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:00:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:00:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:00:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:00:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:00:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:00:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:00:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:00:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:00:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:00:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:00:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:00:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:00:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:00:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:00:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:00:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:00:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:00:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:00:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:00:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:00:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:00:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:00:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:00:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:00:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:00:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:00:57.678 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:00:57.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:00:57.766 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:57.764+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:57.823 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:57.823 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:57.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:00:57.876 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:57.874+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:57.961 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:57.961 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:57.961 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:57.961 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:57.961 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:57.970 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:57.968+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:57.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:00:58.073 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:58.071+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:58.169 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:58.169 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:58.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:58.169 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:00:58.170 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:58.174 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:58.172+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:58.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:00:58.245 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T16:00:58.102+0000 12094640 -1 mon.a@0(leader).osd e45 definitely_dead 0 2024-08-30T16:00:58.277 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:58.275+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:58.351 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:58.351 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:58.351 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:58.351 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:58.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:00:58.378 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:58.376+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:58.481 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:58.479+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:58.533 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:00:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:00:58.534 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:58.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:00:58.585 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:58.582+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:58.683 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:58.682+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:58.734 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:58.734 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:00:58.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:00:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:00:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 97.459594898000006, 2024-08-30T16:00:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:00:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:00:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:00:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:00:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:00:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:00:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:00:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:00:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:00:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:00:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:00:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:00:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:00:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:00:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:00:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:00:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:00:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 97.186667537000005, 2024-08-30T16:00:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:00:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:00:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:00:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:00:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:00:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:00:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:00:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:00:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:00:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:00:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:00:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:00:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:00:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:00:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:00:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:00:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:00:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:00:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:00:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:00:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:00:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:00:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:00:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:00:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:00:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:00:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:00:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:00:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:00:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:00:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:00:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:00:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:00:58.743 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:58.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:00:58.748 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:00:58.749 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:00:58.784 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:58.783+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:58.860 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:00:58.860 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:00:58.888 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:58.884+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:58.889 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:00:58.889 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:00:58.889 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:00:58.889 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:00:58.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:00:58.960 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:00:58.978 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2024-08-30T16:00:58.978 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-30T16:00:58.978 DEBUG:teuthology.orchestra.run.smithi094:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-08-30T16:00:58.981 INFO:tasks.ceph.mon.b:Started 2024-08-30T16:00:58.981 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-08-30T16:00:58.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:00:58.985 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:00:58.986 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:00:58.986 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:58.983+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:59.094 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:59.094 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:59.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:00:59.111 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:00:59.112 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:00:59.112 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:00:59.110+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:00:59.188 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:00:59.186+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:00:59.195 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:00:59.196 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:00:59.246 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:59.246 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:59.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:00:59.283 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:00:59.283 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:00:59.326 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:00:59.324+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:00:59.379 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:00:59.379 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:00:59.404 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:00:59.403+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:00:59.411 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:59.412 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:59.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:00:59.465 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:00:59.466 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:00:59.491 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:00:59.489+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:00:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:marked out osd.6. 2024-08-30T16:00:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-08-30T16:00:59.592 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:00:59.590+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:00:59.619 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:59.619 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:59.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:00:59.693 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:59.691+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:00:59.757 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:00:59.768 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:59.768 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:59.777 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:00:59.820 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:59.817+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:59.894 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:00:59.892+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:00:59.916 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:00:59.916 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:00:59.926 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:00:59.995 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:00:59.993+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:00.052 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:00.052 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:00.061 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:00.096 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:00.094+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:00.197 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:00.195+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:00.228 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:00.229 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:00.237 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:00.322 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:00.320+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:00.399 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:00.398+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:00.404 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:00.404 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:00.414 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:00.500 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:00.498+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:00.583 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:00.583 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:00.592 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:00.637 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:00.635+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:00.702 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:00.699+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:00.736 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:00.736 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:00.746 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:00.831 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:00.829+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:00.903 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:00.902+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:00.973 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:00.973 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:00.982 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:01.004 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:01.002+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:01.224 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:01.222+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:01.326 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:01.324+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:01.351 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:01.351 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:01.361 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:01.426 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:01.423+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:01.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:01:01.526 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:01.524+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:01.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-08-30T16:01:01.553 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:01.554 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:01.563 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:01.627 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:01.625+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:01.691 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:01.691 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:01.700 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:01.728 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:01.726+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:01.793 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:01:01.794 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:01:01.829 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:01.827+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:01.831 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:01.831 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:01.840 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:01.925 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:01:01.925 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:01:01.930 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:01.928+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:02.011 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:02.011 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:02.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:02.021 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:01:02.021 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:01:02.031 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:02.029+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:02.117 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:01:02.117 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:01:02.132 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:02.130+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:02.151 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:02.151 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:02.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:02.216 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:01:02.216 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:01:02.240 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:02.239+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:02.292 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:02.292 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:02.292 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:02.292 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:02.298 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:01:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:01:02.301 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:01:02.345 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:02.343+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:02.381 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:01:02.381 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:01:02.429 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:02.429 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:02.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:02.429 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:01:02.429 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:02.435 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:02.433+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:02.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:01:02.464 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:01:02.464 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:01:02.536 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:02.533+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:02.593 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:02.593 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:02.594 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:02.594 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:02.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:01:02.637 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:02.635+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:02.738 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:02.735+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:02.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:01:02.775 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-08-30T16:01:02.778 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:02.778 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:02.778 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:02.778 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:02.778 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:02.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:01:02.839 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:02.837+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:02.940 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:02.938+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:02.977 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:02.977 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:02.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:02.977 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:02.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.977 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:02.977 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:02.977 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 101.702323423, 2024-08-30T16:01:02.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:01:02.978 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:02.978 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:02.978 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:02.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:02.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:02.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:02.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:02.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:02.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:02.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:02.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:01:02.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:01:02.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:02.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:01:02.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:01:02.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:02.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:01:02.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:01:02.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:02.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:01:02.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:01:02.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:02.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:01:02.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:01:02.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:02.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:01:02.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:01:02.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:02.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:01:02.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:01:02.981 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:02.981 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:02.981 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:02.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.982 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:02.982 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:02.982 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 101.429396062, 2024-08-30T16:01:02.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:01:02.982 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:02.982 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:02.982 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:02.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:02.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:02.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:02.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:02.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:02.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:02.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:02.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:01:02.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:01:02.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:02.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:01:02.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:01:02.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:02.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:01:02.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:01:02.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:02.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:01:02.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:01:02.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:02.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:01:02.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:01:02.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:02.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:01:02.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:01:02.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:02.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:02.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:02.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:01:02.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:01:02.986 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:02.986 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:02.986 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:02.986 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:02.986 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:02.986 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:02.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:01:03.071 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:03.069+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:03.142 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:03.141+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:03.160 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:03.160 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:03.160 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:03.160 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:03.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:01:03.244 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:03.242+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:03.337 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:03.337 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:03.337 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:03.337 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:03.337 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:03.344 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:03.343+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:03.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:01:03.428 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:01:03.445 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:03.443+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:03.477 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:03.477 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:03.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:03.477 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:03.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:03.477 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:01:03.477 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:03.477 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 98.099845477000002, 2024-08-30T16:01:03.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:01:03.478 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:03.478 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:03.478 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:03.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:03.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:03.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:03.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:03.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:03.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:03.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:03.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:03.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:01:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:01:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:01:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:01:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:03.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:03.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:01:03.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:01:03.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:03.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:03.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:03.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:01:03.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:01:03.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:03.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:03.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:03.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:01:03.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:01:03.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:03.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:03.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:03.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:01:03.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:01:03.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:03.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:03.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:03.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:01:03.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:01:03.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:03.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:03.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:03.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:01:03.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:01:03.482 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:03.482 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:03.482 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:03.482 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:03.482 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:03.482 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:03.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:01:03.546 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:03.544+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:03.647 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:03.645+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:03.659 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:03.659 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:03.659 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:03.659 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:03.669 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:01:03.748 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:03.745+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:03.806 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:03.806 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:03.806 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:03.806 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:03.806 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:03.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:01:03.848 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:03.846+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:03.949 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:03.947+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:04.009 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:04.009 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:04.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:04.009 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:04.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.009 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:04.009 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:04.009 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 99.555303362000004, 2024-08-30T16:01:04.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:01:04.009 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:04.010 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:04.010 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:04.010 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:04.010 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:04.010 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:04.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.010 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:04.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:04.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:04.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:04.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:04.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:04.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:01:04.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:01:04.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:04.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:01:04.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:01:04.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:04.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:01:04.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:01:04.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:04.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:01:04.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:01:04.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:04.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:01:04.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:01:04.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:01:04.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:01:04.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:01:04.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:04.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:01:04.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:01:04.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:04.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:01:04.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:01:04.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:01:04.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:01:04.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:01:04.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:04.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:01:04.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:01:04.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:04.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:01:04.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:01:04.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:04.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:01:04.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:01:04.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:04.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:01:04.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:01:04.016 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.016 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:04.016 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.017 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:01:04.017 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:04.017 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 98.439437026999997, 2024-08-30T16:01:04.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:01:04.017 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:04.017 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:04.017 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:04.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:04.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:04.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:01:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:01:04.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:01:04.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:01:04.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:04.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:01:04.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:01:04.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:04.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:01:04.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:01:04.020 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.020 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:04.020 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 98.379845528999994, 2024-08-30T16:01:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:01:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:01:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.021 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:04.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:04.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:04.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:04.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:04.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:04.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:01:04.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:01:04.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:04.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:01:04.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:01:04.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:04.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:01:04.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:01:04.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:04.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:01:04.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:01:04.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:04.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:01:04.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:01:04.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:04.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:01:04.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:01:04.024 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:01:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 98.353146842000001, 2024-08-30T16:01:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:01:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:04.026 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:04.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.026 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:04.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:04.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:04.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:04.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:04.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:01:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:01:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:01:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:01:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:01:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:04.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:01:04.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:01:04.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:04.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:01:04.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:01:04.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:04.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:01:04.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:01:04.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.029 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:04.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.029 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:04.029 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:04.029 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 97.485530314000002, 2024-08-30T16:01:04.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:01:04.029 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:04.029 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:04.030 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:04.030 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:04.030 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:04.030 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:04.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.030 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:04.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:04.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:04.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:04.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:04.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:04.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:01:04.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:04.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:04.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:01:04.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:01:04.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:04.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:01:04.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:01:04.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:04.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:01:04.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:01:04.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:04.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:01:04.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:01:04.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:04.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:01:04.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:01:04.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:04.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:01:04.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:01:04.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:04.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:01:04.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:01:04.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:04.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:01:04.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:01:04.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:04.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:01:04.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:01:04.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:04.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:01:04.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:01:04.035 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.035 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:04.035 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.035 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:01:04.035 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:04.035 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 97.326283235000005, 2024-08-30T16:01:04.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:01:04.036 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:04.036 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:04.036 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:04.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:04.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:04.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:04.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:04.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:01:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:01:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:01:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:01:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:04.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:01:04.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:01:04.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:04.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:01:04.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:01:04.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:04.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:01:04.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:01:04.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:04.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:01:04.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:01:04.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:04.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:01:04.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:01:04.039 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.039 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:04.039 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.040 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:04.040 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:04.040 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 97.050109168999995, 2024-08-30T16:01:04.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:01:04.040 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:04.040 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:04.040 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:04.040 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:04.040 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:04.040 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:04.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.041 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:04.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:04.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:04.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:04.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:04.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:04.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:01:04.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:04.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:04.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:01:04.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:01:04.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:04.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:01:04.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:01:04.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:04.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:01:04.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:01:04.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:04.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:01:04.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:01:04.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:04.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:01:04.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:01:04.044 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.044 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:04.044 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 96.391817590000002, 2024-08-30T16:01:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:01:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:01:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.045 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:04.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:04.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:04.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:04.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:04.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:04.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:01:04.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:04.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:04.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:01:04.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:01:04.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:04.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:01:04.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:01:04.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:04.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:01:04.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:01:04.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:04.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:01:04.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:01:04.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:04.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:01:04.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:01:04.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:04.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:01:04.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:01:04.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:04.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:01:04.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:01:04.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:01:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:01:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:01:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:01:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:01:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:01:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:01:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 96.295904305999997, 2024-08-30T16:01:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:01:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:04.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:04.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:04.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:04.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:04.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:01:04.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:04.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:04.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:01:04.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:01:04.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:04.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:01:04.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:01:04.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:04.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:01:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:01:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:01:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:01:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:01:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:01:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:04.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:01:04.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:01:04.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.055 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:04.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.055 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:04.055 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:04.055 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 37.044543795000003, 2024-08-30T16:01:04.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:01:04.055 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:04.056 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:04.056 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:04.056 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:04.056 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:04.056 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:04.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.056 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:04.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:04.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:04.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:04.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:04.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:04.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:04.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:01:04.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:01:04.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:04.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:01:04.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:01:04.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:04.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:01:04.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:01:04.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:04.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:01:04.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:01:04.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:04.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:01:04.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:01:04.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:04.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:04.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:04.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:01:04.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:01:04.059 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.059 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:04.059 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.059 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:04.059 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:04.059 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:04.061 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:01:04.062 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:04.059+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:04.162 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:04.160+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:04.187 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:04.187 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:04.187 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:04.197 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:01:04.283 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:04.281+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:04.364 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:04.362+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:04.365 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:04.365 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:04.365 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:04.365 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:04.366 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:04.375 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:04.465 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:04.463+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:04.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:01:04.545 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:01:04.546 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:04.551 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:04.553 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:01:04.553 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:01:04.553 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-30T16:01:04.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-08-30T16:01:04.594 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:04.591+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:04.667 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:04.665+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:04.718 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:04.718 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:04.718 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:04.727 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:04.733 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:01:04.733 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:01:04.769 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:04.767+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:04.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-08-30T16:01:04.827 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:01:04.827 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:01:04.869 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:04.866+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:04.893 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:04.893 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:04.893 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:04.893 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:04.893 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:04.903 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:04.926 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:01:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:01:04.969 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:04.968+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:05.021 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:01:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:01:05.071 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:05.068+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:05.080 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:05.080 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:05.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:05.081 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:05.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.081 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:05.081 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:05.081 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 114.216675142, 2024-08-30T16:01:05.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:01:05.081 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.081 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.081 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.081 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.081 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.081 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:01:05.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.082 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:05.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:05.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:01:05.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:01:05.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:01:05.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:01:05.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:01:05.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:01:05.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:01:05.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:01:05.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:01:05.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:01:05.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:01:05.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:01:05.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:01:05.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:01:05.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:01:05.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:01:05.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:01:05.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:01:05.087 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.087 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.087 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.087 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:05.087 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:05.087 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 113.300259122, 2024-08-30T16:01:05.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:01:05.087 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.087 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.087 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.087 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.088 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.088 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:01:05.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.088 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:05.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:05.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:01:05.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:05.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:01:05.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:01:05.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:01:05.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:01:05.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:01:05.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:01:05.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:01:05.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:01:05.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:01:05.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:01:05.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:01:05.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:01:05.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:01:05.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:01:05.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:01:05.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:01:05.092 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.092 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.092 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.092 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:05.092 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:05.093 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 113.298554569, 2024-08-30T16:01:05.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:01:05.093 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.093 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.093 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.093 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.093 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.093 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:01:05.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.093 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:01:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:01:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:01:05.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:01:05.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:01:05.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:01:05.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:01:05.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:01:05.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:01:05.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:01:05.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:01:05.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:01:05.097 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.097 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.097 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.098 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.098 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:05.098 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.87910076199999, 2024-08-30T16:01:05.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:01:05.098 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.098 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.098 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.098 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.098 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.098 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:01:05.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.098 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:05.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:05.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:01:05.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:01:05.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:01:05.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:01:05.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:01:05.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:01:05.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:01:05.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:01:05.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:01:05.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:01:05.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:01:05.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:01:05.102 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.103 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.103 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.103 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.103 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:05.103 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.86221258800001, 2024-08-30T16:01:05.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:01:05.103 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.103 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.103 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.103 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.103 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.104 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:01:05.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.104 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:05.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:05.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:01:05.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:01:05.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:01:05.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:01:05.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:01:05.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:01:05.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:01:05.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:01:05.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:01:05.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:01:05.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:01:05.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:01:05.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:01:05.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:01:05.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:01:05.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:01:05.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:01:05.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:01:05.108 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.108 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.108 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.108 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.108 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:05.108 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.84921251900001, 2024-08-30T16:01:05.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:01:05.108 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.109 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.109 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.109 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.109 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.109 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:01:05.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.109 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:05.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:05.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:01:05.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:01:05.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:01:05.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:01:05.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:01:05.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:01:05.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:01:05.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:01:05.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:01:05.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:01:05.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:01:05.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:01:05.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:01:05.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:01:05.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:01:05.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:01:05.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:01:05.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:01:05.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.113 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.113 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.114 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.114 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:05.114 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.84672022700001, 2024-08-30T16:01:05.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:01:05.114 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.114 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.114 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.114 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.114 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.114 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:01:05.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.114 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:05.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:05.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:01:05.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:05.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:01:05.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:01:05.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:01:05.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:01:05.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:01:05.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:01:05.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:01:05.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:01:05.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:01:05.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:01:05.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:01:05.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:01:05.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:01:05.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:01:05.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:01:05.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:01:05.119 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.119 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.119 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.119 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.119 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:05.119 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.845962779, 2024-08-30T16:01:05.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:01:05.119 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.119 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.119 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.119 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.120 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.120 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:01:05.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.120 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:05.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:05.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:01:05.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:01:05.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:01:05.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:01:05.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:01:05.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:01:05.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:01:05.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:01:05.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:01:05.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:01:05.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:01:05.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:01:05.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:01:05.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:01:05.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:01:05.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:01:05.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:01:05.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:01:05.124 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.124 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.124 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.124 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.124 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:05.124 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.845230329, 2024-08-30T16:01:05.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:01:05.125 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.125 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.125 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.125 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.125 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.125 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:01:05.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.125 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:05.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:05.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:01:05.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:05.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:01:05.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:01:05.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:01:05.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:01:05.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:01:05.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:01:05.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:01:05.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:01:05.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:01:05.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:01:05.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:01:05.129 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.129 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.129 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.129 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.129 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:05.130 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.84269153, 2024-08-30T16:01:05.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:01:05.130 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.130 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.130 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.130 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.130 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.130 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:01:05.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.130 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:05.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:05.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:01:05.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:05.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:01:05.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:01:05.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:01:05.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:01:05.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:01:05.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:01:05.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:01:05.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:01:05.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:01:05.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:01:05.135 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.135 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.135 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.135 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.135 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:05.135 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.84194352900001, 2024-08-30T16:01:05.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:01:05.135 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.135 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.135 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:01:05.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:01:05.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:01:05.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:01:05.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:01:05.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:01:05.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:01:05.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:01:05.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:01:05.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:01:05.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:01:05.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:01:05.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:01:05.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:01:05.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:01:05.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:01:05.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:01:05.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:01:05.140 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.140 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.140 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.140 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.140 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:05.140 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.069827057, 2024-08-30T16:01:05.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:01:05.140 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.140 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.141 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.141 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.141 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.141 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:01:05.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.141 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:05.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:05.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:01:05.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:01:05.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:01:05.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:01:05.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:01:05.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:01:05.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:01:05.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:01:05.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:01:05.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:01:05.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:01:05.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:01:05.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:01:05.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:01:05.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:01:05.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:01:05.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:01:05.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:01:05.145 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.145 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.145 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.145 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.145 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:05.146 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 112.067769911, 2024-08-30T16:01:05.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:01:05.146 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.146 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.146 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.146 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.146 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.146 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:01:05.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.146 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:05.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:05.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:01:05.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:05.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:01:05.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:01:05.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:01:05.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:01:05.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:01:05.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:01:05.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:01:05.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:01:05.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:01:05.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:01:05.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:01:05.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:01:05.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:01:05.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:01:05.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:01:05.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:01:05.150 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.150 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 111.804825411, 2024-08-30T16:01:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:01:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:01:05.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:01:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:01:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:01:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:01:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:01:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:01:05.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:01:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:01:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:01:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:01:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:01:05.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:01:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:01:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:01:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:01:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:01:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:01:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:01:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 111.803733908, 2024-08-30T16:01:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:01:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.156 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:01:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:05.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:01:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:01:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:01:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:01:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:01:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:01:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:01:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:01:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:01:05.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:01:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:01:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:01:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:01:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:01:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:01:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:01:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:01:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.161 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 111.802984323, 2024-08-30T16:01:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:01:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:01:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:01:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:05.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:01:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:01:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:01:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:01:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:01:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:01:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:01:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:01:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:01:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:01:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:01:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:01:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:01:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:01:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:01:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:01:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.166 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 111.769252791, 2024-08-30T16:01:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:01:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.167 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:01:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:01:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:01:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:01:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:01:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:01:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:01:05.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:01:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:01:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:01:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:01:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:01:05.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:01:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:01:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:01:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:01:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:01:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:01:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:01:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 111.76872678300001, 2024-08-30T16:01:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:01:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.172 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:01:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:05.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:01:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:01:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:01:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:01:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:01:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:01:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:01:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:01:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:01:05.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:01:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:01:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:01:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:01:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:01:05.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:01:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:01:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:01:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:05.177 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 111.76827122, 2024-08-30T16:01:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:01:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:01:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:01:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:01:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:01:05.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:01:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:01:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:01:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:01:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:01:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:01:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:01:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:01:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:01:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:01:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:01:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:01:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:01:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:01:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:01:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.182 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:01:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.097755904, 2024-08-30T16:01:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:01:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:01:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:05.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:01:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:01:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:01:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:01:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:01:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:01:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:01:05.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:01:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:01:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:01:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:01:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:01:05.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:01:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:01:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:01:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:01:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:01:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:05.188 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:05.188 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:05.189 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:01:05.189 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:01:05.190 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:05.190 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:05.188+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:05.199 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:01:05.199 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:01:05.291 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:05.288+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:05.312 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:01:05.312 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:01:05.356 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:05.357 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:05.365 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:05.410 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:05.408+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:05.410 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:01:05.410 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:01:05.428 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:01:05.428 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:05.428 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 48, 2024-08-30T16:01:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "9d72e2e2-0554-42af-8e62-9b1a3a804112", 2024-08-30T16:01:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-08-30T15:58:20.129410+0000", 2024-08-30T16:01:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-08-30T16:01:02.575922+0000", 2024-08-30T16:01:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "last_up_change": "2024-08-30T15:59:19.280372+0000", 2024-08-30T16:01:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "last_in_change": "2024-08-30T16:00:59.018230+0000", 2024-08-30T16:01:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T16:01:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_num": 5799936, 2024-08-30T16:01:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_set": [ 2024-08-30T16:01:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "pglog_hardlimit", 2024-08-30T16:01:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "purged_snapdirs", 2024-08-30T16:01:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_deletes", 2024-08-30T16:01:05.429 INFO:teuthology.orchestra.run.smithi050.stdout: "sortbitwise" 2024-08-30T16:01:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:01:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_version": 7, 2024-08-30T16:01:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T16:01:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T16:01:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T16:01:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_snapshot": "", 2024-08-30T16:01:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_max": 2, 2024-08-30T16:01:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd": 8, 2024-08-30T16:01:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "require_min_compat_client": "luminous", 2024-08-30T16:01:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "min_compat_client": "luminous", 2024-08-30T16:01:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "require_osd_release": "reef", 2024-08-30T16:01:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_crimson": false, 2024-08-30T16:01:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: "pools": [ 2024-08-30T16:01:05.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 1, 2024-08-30T16:01:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".mgr", 2024-08-30T16:01:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-08-30T15:59:08.355540+0000", 2024-08-30T16:01:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-08-30T16:01:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-08-30T16:01:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-08-30T16:01:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 2, 2024-08-30T16:01:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 1, 2024-08-30T16:01:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-08-30T16:01:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-08-30T16:01:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-08-30T16:01:05.431 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T16:01:05.432 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T16:01:05.432 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-08-30T16:01:05.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-08-30T16:01:05.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 1, 2024-08-30T16:01:05.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 1, 2024-08-30T16:01:05.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 1, 2024-08-30T16:01:05.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 1, 2024-08-30T16:01:05.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 1, 2024-08-30T16:01:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-08-30T16:01:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-08-30T16:01:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-08-30T16:01:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-08-30T16:01:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-08-30T16:01:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-08-30T16:01:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-08-30T16:01:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "19", 2024-08-30T16:01:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-08-30T16:01:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T16:01:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T16:01:05.433 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-08-30T16:01:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-08-30T16:01:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-08-30T16:01:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-08-30T16:01:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-08-30T16:01:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-08-30T16:01:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-08-30T16:01:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-08-30T16:01:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-08-30T16:01:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-08-30T16:01:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-08-30T16:01:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-08-30T16:01:05.434 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-08-30T16:01:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-08-30T16:01:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-08-30T16:01:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T16:01:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T16:01:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T16:01:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-08-30T16:01:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-08-30T16:01:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-08-30T16:01:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-08-30T16:01:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-08-30T16:01:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-08-30T16:01:05.435 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-08-30T16:01:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-08-30T16:01:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-08-30T16:01:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-08-30T16:01:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T16:01:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-08-30T16:01:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-08-30T16:01:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-08-30T16:01:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-08-30T16:01:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-08-30T16:01:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-08-30T16:01:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_max": 32, 2024-08-30T16:01:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_min": 1 2024-08-30T16:01:05.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-08-30T16:01:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": {} 2024-08-30T16:01:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-08-30T16:01:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 8, 2024-08-30T16:01:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 8, 2024-08-30T16:01:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.25, 2024-08-30T16:01:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-08-30T16:01:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-08-30T16:01:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-08-30T16:01:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-08-30T16:01:05.437 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-08-30T16:01:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-08-30T16:01:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "rbd", 2024-08-30T16:01:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-08-30T15:59:21.696796+0000", 2024-08-30T16:01:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-08-30T16:01:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-08-30T16:01:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-08-30T16:01:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 2, 2024-08-30T16:01:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 1, 2024-08-30T16:01:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-08-30T16:01:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-08-30T16:01:05.438 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-08-30T16:01:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T16:01:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T16:01:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-08-30T16:01:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-08-30T16:01:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 8, 2024-08-30T16:01:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 8, 2024-08-30T16:01:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 8, 2024-08-30T16:01:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 8, 2024-08-30T16:01:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 8, 2024-08-30T16:01:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-08-30T16:01:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-08-30T16:01:05.439 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-08-30T16:01:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-08-30T16:01:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-08-30T16:01:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-08-30T16:01:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-08-30T16:01:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "28", 2024-08-30T16:01:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-08-30T16:01:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T16:01:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T16:01:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-08-30T16:01:05.440 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-08-30T16:01:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 3, 2024-08-30T16:01:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 28, 2024-08-30T16:01:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-08-30T16:01:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-08-30T16:01:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-08-30T16:01:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-08-30T16:01:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-08-30T16:01:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-08-30T16:01:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-08-30T16:01:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-08-30T16:01:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-08-30T16:01:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-08-30T16:01:05.441 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-08-30T16:01:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T16:01:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T16:01:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T16:01:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-08-30T16:01:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-08-30T16:01:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-08-30T16:01:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-08-30T16:01:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-08-30T16:01:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-08-30T16:01:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-08-30T16:01:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-08-30T16:01:05.442 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-08-30T16:01:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-08-30T16:01:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T16:01:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-08-30T16:01:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-08-30T16:01:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-08-30T16:01:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-08-30T16:01:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-08-30T16:01:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-08-30T16:01:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-08-30T16:01:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd": {} 2024-08-30T16:01:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.443 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-08-30T16:01:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 4.9800000190734863, 2024-08-30T16:01:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 4.9800000190734863, 2024-08-30T16:01:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.87999999523162842, 2024-08-30T16:01:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 4.380000114440918, 2024-08-30T16:01:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 4.380000114440918, 2024-08-30T16:01:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-08-30T16:01:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-08-30T16:01:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-08-30T16:01:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:01:05.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-08-30T16:01:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-08-30T16:01:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4", 2024-08-30T16:01:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:01:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:01:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:01:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:01:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:01:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:01:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 15, 2024-08-30T16:01:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:01:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:01:05.445 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:01:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:01:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6812", 2024-08-30T16:01:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:01:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:01:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.446 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6813", 2024-08-30T16:01:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:01:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:01:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6815", 2024-08-30T16:01:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:01:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.447 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:01:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6814", 2024-08-30T16:01:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:01:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6812/1957399494", 2024-08-30T16:01:05.448 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6813/1957399494", 2024-08-30T16:01:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6815/1957399494", 2024-08-30T16:01:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6814/1957399494", 2024-08-30T16:01:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:01:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:01:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:01:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-08-30T16:01:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "0fe11a1c-4326-4bfd-a2a4-bff14023f301", 2024-08-30T16:01:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:01:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:01:05.449 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:01:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:01:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:01:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:01:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 22, 2024-08-30T16:01:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 22, 2024-08-30T16:01:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:01:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:01:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:01:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6808", 2024-08-30T16:01:05.450 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:01:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:01:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6809", 2024-08-30T16:01:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:01:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.451 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:01:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6811", 2024-08-30T16:01:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:01:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:01:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.452 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6810", 2024-08-30T16:01:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:01:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6808/3208178756", 2024-08-30T16:01:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6809/3208178756", 2024-08-30T16:01:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6811/3208178756", 2024-08-30T16:01:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6810/3208178756", 2024-08-30T16:01:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:01:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:01:05.453 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:01:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-08-30T16:01:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "a9d5354b-3600-41ed-80c5-a3439da7aca6", 2024-08-30T16:01:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:01:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:01:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:01:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:01:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:01:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:01:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 16, 2024-08-30T16:01:05.454 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:01:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:01:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:01:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:01:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6804", 2024-08-30T16:01:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:01:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.455 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:01:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6805", 2024-08-30T16:01:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:01:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:01:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.456 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6807", 2024-08-30T16:01:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:01:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:01:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6806", 2024-08-30T16:01:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:01:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6804/287018800", 2024-08-30T16:01:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6805/287018800", 2024-08-30T16:01:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6807/287018800", 2024-08-30T16:01:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6806/287018800", 2024-08-30T16:01:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:01:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:01:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:01:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-08-30T16:01:05.458 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "78bcc398-a7e4-4efc-9836-1dfb94407c9c", 2024-08-30T16:01:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:01:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:01:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:01:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:01:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:01:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:01:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:01:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 46, 2024-08-30T16:01:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:01:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:01:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:01:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6800", 2024-08-30T16:01:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:01:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:01:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6801", 2024-08-30T16:01:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:01:05.460 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:01:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6803", 2024-08-30T16:01:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:01:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.461 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:01:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6802", 2024-08-30T16:01:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:01:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6800/1140717714", 2024-08-30T16:01:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6801/1140717714", 2024-08-30T16:01:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6803/1140717714", 2024-08-30T16:01:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6802/1140717714", 2024-08-30T16:01:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:01:05.462 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:01:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:01:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-08-30T16:01:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "53a1060b-7985-4c2c-820e-64b158056113", 2024-08-30T16:01:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:01:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:01:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:01:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:01:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:01:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:01:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 20, 2024-08-30T16:01:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 24, 2024-08-30T16:01:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:01:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:01:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:01:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6805", 2024-08-30T16:01:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:01:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:01:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6806", 2024-08-30T16:01:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:01:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:01:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6808", 2024-08-30T16:01:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:01:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:01:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6807", 2024-08-30T16:01:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:01:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.466 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6805/1468073345", 2024-08-30T16:01:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6806/1468073345", 2024-08-30T16:01:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6808/1468073345", 2024-08-30T16:01:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6807/1468073345", 2024-08-30T16:01:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:01:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:01:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:01:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-08-30T16:01:05.467 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "853a03f7-227d-4f98-9e5c-c29896820746", 2024-08-30T16:01:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:01:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:01:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:01:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:01:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:01:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:01:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 13, 2024-08-30T16:01:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 46, 2024-08-30T16:01:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:01:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:01:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:01:05.468 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6809", 2024-08-30T16:01:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:01:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:01:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6810", 2024-08-30T16:01:05.469 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:01:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:01:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6812", 2024-08-30T16:01:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:01:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:01:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6811", 2024-08-30T16:01:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:01:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6809/3876025616", 2024-08-30T16:01:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6810/3876025616", 2024-08-30T16:01:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6812/3876025616", 2024-08-30T16:01:05.471 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6811/3876025616", 2024-08-30T16:01:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:01:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:01:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:01:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-08-30T16:01:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "bc978060-bf28-4027-8bba-7d6d8659bfc5", 2024-08-30T16:01:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:01:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:01:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:01:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:01:05.472 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:01:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:01:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:01:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:01:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:01:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:01:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:01:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6801", 2024-08-30T16:01:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:01:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:01:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6802", 2024-08-30T16:01:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:01:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:01:05.474 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6804", 2024-08-30T16:01:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:01:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:01:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6803", 2024-08-30T16:01:05.475 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:01:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6801/128519122", 2024-08-30T16:01:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6802/128519122", 2024-08-30T16:01:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6804/128519122", 2024-08-30T16:01:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6803/128519122", 2024-08-30T16:01:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:01:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:01:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:01:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-08-30T16:01:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "45df5c6f-b124-46d1-9efb-62b78bc584ed", 2024-08-30T16:01:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:01:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:01:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:01:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:01:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:01:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:01:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:01:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 40, 2024-08-30T16:01:05.477 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:01:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:01:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:01:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6813", 2024-08-30T16:01:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:01:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:01:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.478 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6814", 2024-08-30T16:01:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:01:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:01:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6816", 2024-08-30T16:01:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:01:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.479 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:01:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:01:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:01:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6815", 2024-08-30T16:01:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:01:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6813/68721277", 2024-08-30T16:01:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6814/68721277", 2024-08-30T16:01:05.480 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6816/68721277", 2024-08-30T16:01:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6815/68721277", 2024-08-30T16:01:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:01:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:01:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:01:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:01:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_xinfo": [ 2024-08-30T16:01:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-08-30T16:01:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:01:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:01:05.481 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:01:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:01:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:01:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.911895+0000", 2024-08-30T16:01:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:01:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-08-30T16:01:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:01:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:01:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:01:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:01:05.482 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:01:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:04.786543+0000", 2024-08-30T16:01:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:01:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-08-30T16:01:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:01:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:01:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:01:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:01:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:01:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:04.197952+0000", 2024-08-30T16:01:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:01:05.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-08-30T16:01:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:01:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:01:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:01:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:01:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:01:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.680847+0000", 2024-08-30T16:01:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:01:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-08-30T16:01:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:01:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:01:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:01:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:01:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:01:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:01.937600+0000", 2024-08-30T16:01:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:01:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-08-30T16:01:05.485 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:01:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:01:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:01:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:01:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:01:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:02.082761+0000", 2024-08-30T16:01:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:01:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-08-30T16:01:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:01:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:01:05.486 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:01:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:01:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:01:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.401025+0000", 2024-08-30T16:01:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:01:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-08-30T16:01:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:01:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:01:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:01:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:01:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:01:05.487 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.129371+0000", 2024-08-30T16:01:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:01:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:01:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap": [ 2024-08-30T16:01:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "2.3", 2024-08-30T16:01:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-08-30T16:01:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: 7, 2024-08-30T16:01:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: 6 2024-08-30T16:01:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.488 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:01:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_items": [], 2024-08-30T16:01:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_primaries": [], 2024-08-30T16:01:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_temp": [], 2024-08-30T16:01:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_temp": [], 2024-08-30T16:01:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklist": {}, 2024-08-30T16:01:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "range_blocklist": {}, 2024-08-30T16:01:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profiles": { 2024-08-30T16:01:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "default": { 2024-08-30T16:01:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-08-30T16:01:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-08-30T16:01:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-08-30T16:01:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-08-30T16:01:05.489 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van" 2024-08-30T16:01:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps_queue": [], 2024-08-30T16:01:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "new_removed_snaps": [], 2024-08-30T16:01:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "new_purged_snaps": [], 2024-08-30T16:01:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_node_flags": {}, 2024-08-30T16:01:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "device_class_flags": {}, 2024-08-30T16:01:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": { 2024-08-30T16:01:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_enabled": false, 2024-08-30T16:01:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_bucket_count": 0, 2024-08-30T16:01:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "degraded_stretch_mode": 0, 2024-08-30T16:01:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "recovering_stretch_mode": 0, 2024-08-30T16:01:05.490 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_bucket": 0 2024-08-30T16:01:05.491 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:05.491 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:05.492 INFO:tasks.thrashosds.thrasher:j is {'epoch': 48, 'fsid': '9d72e2e2-0554-42af-8e62-9b1a3a804112', 'created': '2024-08-30T15:58:20.129410+0000', 'modified': '2024-08-30T16:01:02.575922+0000', 'last_up_change': '2024-08-30T15:59:19.280372+0000', 'last_in_change': '2024-08-30T16:00:59.018230+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T15:59:08.355540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-08-30T15:59:21.696796+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6812', 'nonce': 1957399494}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6813', 'nonce': 1957399494}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 1957399494}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6814', 'nonce': 1957399494}]}, 'public_addr': '172.21.15.50:6812/1957399494', 'cluster_addr': '172.21.15.50:6813/1957399494', 'heartbeat_back_addr': '172.21.15.50:6815/1957399494', 'heartbeat_front_addr': '172.21.15.50:6814/1957399494', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0fe11a1c-4326-4bfd-a2a4-bff14023f301', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6808', 'nonce': 3208178756}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6809', 'nonce': 3208178756}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 3208178756}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6810', 'nonce': 3208178756}]}, 'public_addr': '172.21.15.50:6808/3208178756', 'cluster_addr': '172.21.15.50:6809/3208178756', 'heartbeat_back_addr': '172.21.15.50:6811/3208178756', 'heartbeat_front_addr': '172.21.15.50:6810/3208178756', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a9d5354b-3600-41ed-80c5-a3439da7aca6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6804', 'nonce': 287018800}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6805', 'nonce': 287018800}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 287018800}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6806', 'nonce': 287018800}]}, 'public_addr': '172.21.15.50:6804/287018800', 'cluster_addr': '172.21.15.50:6805/287018800', 'heartbeat_back_addr': '172.21.15.50:6807/287018800', 'heartbeat_front_addr': '172.21.15.50:6806/287018800', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '78bcc398-a7e4-4efc-9836-1dfb94407c9c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6800', 'nonce': 1140717714}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6801', 'nonce': 1140717714}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 1140717714}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6802', 'nonce': 1140717714}]}, 'public_addr': '172.21.15.50:6800/1140717714', 'cluster_addr': '172.21.15.50:6801/1140717714', 'heartbeat_back_addr': '172.21.15.50:6803/1140717714', 'heartbeat_front_addr': '172.21.15.50:6802/1140717714', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '53a1060b-7985-4c2c-820e-64b158056113', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 1468073345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6806', 'nonce': 1468073345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6808', 'nonce': 1468073345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 1468073345}]}, 'public_addr': '172.21.15.94:6805/1468073345', 'cluster_addr': '172.21.15.94:6806/1468073345', 'heartbeat_back_addr': '172.21.15.94:6808/1468073345', 'heartbeat_front_addr': '172.21.15.94:6807/1468073345', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '853a03f7-227d-4f98-9e5c-c29896820746', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 3876025616}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6810', 'nonce': 3876025616}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6812', 'nonce': 3876025616}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 3876025616}]}, 'public_addr': '172.21.15.94:6809/3876025616', 'cluster_addr': '172.21.15.94:6810/3876025616', 'heartbeat_back_addr': '172.21.15.94:6812/3876025616', 'heartbeat_front_addr': '172.21.15.94:6811/3876025616', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bc978060-bf28-4027-8bba-7d6d8659bfc5', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 128519122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6802', 'nonce': 128519122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6804', 'nonce': 128519122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 128519122}]}, 'public_addr': '172.21.15.94:6801/128519122', 'cluster_addr': '172.21.15.94:6802/128519122', 'heartbeat_back_addr': '172.21.15.94:6804/128519122', 'heartbeat_front_addr': '172.21.15.94:6803/128519122', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '45df5c6f-b124-46d1-9efb-62b78bc584ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 68721277}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6814', 'nonce': 68721277}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6816', 'nonce': 68721277}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 68721277}]}, 'public_addr': '172.21.15.94:6813/68721277', 'cluster_addr': '172.21.15.94:6814/68721277', 'heartbeat_back_addr': '172.21.15.94:6816/68721277', 'heartbeat_front_addr': '172.21.15.94:6815/68721277', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.911895+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:04.786543+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:04.197952+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.680847+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:01.937600+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:02.082761+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.401025+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.129371+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.3', 'osds': [7, 6]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T16:01:05.493 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.3 2024-08-30T16:01:05.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap 2.3 2024-08-30T16:01:05.493 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:05.493 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:05.494 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:05.490+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:05.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:05.593 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:05.591+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:05.669 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:05.670 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:05.679 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:05.721 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:05.719+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:05.795 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:05.792+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:05.846 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:05.846 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:05.855 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:05.895 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:05.893+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:05.981 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:05.981 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:05.990 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:06.034 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:06.032+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:06.098 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:06.096+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:06.129 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:06.129 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:06.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:06.201 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:06.200+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:06.284 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:06.285 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:06.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:06.302 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:06.300+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:06.397 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-08-30T16:01:06.403 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:06.401+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:06.408 INFO:teuthology.orchestra.run.smithi050.stderr:clear 2.3 pg_upmap mapping 2024-08-30T16:01:06.432 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:06.432 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:06.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:01:06.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:01:06.504 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:06.502+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:06.558 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:07.503 31218== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:06.605 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:06.603+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:06.627 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:06.627 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:06.638 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:01:06.706 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:06.703+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:06.769 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:06.769 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:06.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:01:06.785 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:07.730 31218== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:06.819 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:06.817+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:06.907 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:06.906+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:06.946 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:06.947 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:06.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:01:07.009 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:07.007+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:07.084 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:07.084 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:07.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:01:07.109 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:07.108+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:07.129 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:01:07.211 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:07.208+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:07.222 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:07.222 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:07.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:01:07.316 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:07.315+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:07.399 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:07.399 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:07.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:01:07.424 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:01:07.449 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:07.448+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:07.513 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:07.512+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:07.582 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:07.582 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:07.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:01:07.633 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:07.632+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:07.690 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:01:07.690 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:01:07.715 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:07.713+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:07.770 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:07.771 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:07.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:01:07.791 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:01:07.791 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:01:07.816 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:07.815+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:07.890 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:01:07.890 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:01:07.917 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:07.915+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:07.930 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:07.930 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:07.942 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:01:07.999 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:01:08.000 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:01:08.018 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:08.016+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:08.083 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:01:08.083 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:01:08.119 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:08.117+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:08.145 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:08.145 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:08.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:08.145 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:08.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 103.683132801, 2024-08-30T16:01:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:01:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:08.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:08.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:08.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:08.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:08.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:01:08.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:01:08.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:08.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:01:08.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:01:08.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:08.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:01:08.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:01:08.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:08.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:01:08.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:01:08.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:08.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:01:08.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:01:08.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:01:08.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:01:08.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:01:08.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:08.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:01:08.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:01:08.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:08.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:01:08.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:01:08.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:01:08.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:01:08.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:01:08.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:08.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:01:08.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:01:08.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:08.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:01:08.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:01:08.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:08.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:01:08.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:01:08.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:08.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:01:08.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:01:08.153 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.153 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:08.153 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.153 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:01:08.153 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:08.153 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 102.56726646600001, 2024-08-30T16:01:08.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:01:08.153 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:01:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:01:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:01:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:01:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:01:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:01:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:01:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:01:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:01:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:01:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:01:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:08.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:01:08.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:01:08.158 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.158 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:08.158 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.158 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:08.158 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:08.158 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 102.507674968, 2024-08-30T16:01:08.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:01:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:01:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:08.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:08.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:08.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:01:08.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:01:08.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:08.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:01:08.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:01:08.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:08.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:01:08.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:01:08.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:08.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:01:08.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:01:08.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:08.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:01:08.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:01:08.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:08.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:01:08.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:01:08.162 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.162 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:08.162 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.163 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:01:08.163 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:08.163 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 102.480976281, 2024-08-30T16:01:08.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:01:08.163 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:08.163 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:08.163 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:08.163 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:08.163 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:08.164 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:08.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.164 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:08.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:08.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:08.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:08.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:08.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:08.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:01:08.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:08.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:08.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:01:08.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:01:08.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:08.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:01:08.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:01:08.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:08.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:01:08.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:01:08.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:08.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:01:08.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:01:08.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:08.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:01:08.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:01:08.167 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.167 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:08.167 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.168 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:08.168 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:08.168 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 101.613359753, 2024-08-30T16:01:08.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:01:08.168 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:08.168 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:08.168 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:08.168 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:08.168 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:08.168 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:08.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.168 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:08.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:01:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:08.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:08.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:01:08.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:01:08.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:08.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:01:08.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:01:08.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:01:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:01:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:01:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:01:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:01:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:01:08.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:01:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:01:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:01:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:01:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:01:08.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:01:08.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:08.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:01:08.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:01:08.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:08.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:01:08.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:01:08.174 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.174 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:08.174 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.174 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:01:08.174 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:08.174 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 101.454112674, 2024-08-30T16:01:08.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:01:08.174 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:08.174 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:08.174 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:08.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:08.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:08.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:08.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:08.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:08.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:01:08.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:01:08.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:08.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:01:08.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:01:08.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:01:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:01:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:01:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:01:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:01:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:01:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:08.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:01:08.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:01:08.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:08.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:01:08.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:01:08.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.178 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:08.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.179 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:08.179 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:08.179 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 101.17793860800001, 2024-08-30T16:01:08.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:01:08.179 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:08.179 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:08.179 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:08.179 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:08.179 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:08.179 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:08.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.179 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:08.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:08.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:08.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:08.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:08.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:08.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:01:08.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:08.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:08.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:01:08.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:01:08.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:08.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:01:08.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:01:08.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:08.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:01:08.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:01:08.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:08.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:01:08.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:01:08.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:08.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:01:08.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:01:08.184 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.184 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:08.184 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.184 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:08.184 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:08.184 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 100.519647029, 2024-08-30T16:01:08.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:01:08.184 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:08.184 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:08.184 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:08.185 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:08.185 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:08.185 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:01:08.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.185 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:08.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:08.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:08.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:01:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:01:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:01:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:01:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:01:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:01:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:01:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:01:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:01:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:01:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:01:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:01:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:01:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:01:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:01:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:01:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:01:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:01:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:01:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:01:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:01:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:01:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 100.42373374500001, 2024-08-30T16:01:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:01:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:01:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:01:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:01:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:01:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:01:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:08.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:01:08.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:01:08.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:08.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:01:08.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:01:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:01:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:01:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:01:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:01:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.196 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:08.196 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:08.196 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 41.172373233999998, 2024-08-30T16:01:08.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:01:08.196 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:08.196 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:08.196 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:08.196 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:08.197 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:08.197 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:08.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.197 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:08.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:08.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:08.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:01:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:01:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:01:08.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:01:08.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:08.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:01:08.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:01:08.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:08.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:01:08.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:01:08.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:08.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:01:08.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:01:08.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:08.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:08.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:08.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:01:08.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:01:08.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.200 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:08.201 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.201 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:08.201 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:08.201 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:08.202 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:01:08.203 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:01:08.203 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:01:08.247 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:08.245+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:08.283 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:01:08.284 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:01:08.321 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:08.320+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:08.347 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:08.348 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:08.348 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:08.357 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:01:08.368 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:01:08.369 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:01:08.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-08-30T16:01:08.443 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:08.441+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:08.523 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:08.521+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:08.534 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:08.534 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:08.534 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:08.534 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:08.534 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:08.543 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:08.623 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:08.622+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:08.710 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:08.710 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:08.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:08.711 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:01:08.711 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:08.720 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:08.764 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:08.762+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:08.825 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:08.823+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:08.846 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:08.846 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:08.846 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:08.855 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:08.926 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:08.924+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:09.025 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:09.026 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:09.026 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:09.026 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:09.026 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:09.026 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:09.025+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:09.034 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:09.126 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:09.125+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:09.217 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:09.218 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.353224724, 2024-08-30T16:01:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:01:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.219 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:01:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.220 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:09.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:01:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:01:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:01:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:01:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:01:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:01:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:01:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:01:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:01:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:01:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:01:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:01:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:01:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:01:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:01:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:01:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:01:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:01:09.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:01:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:01:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:01:09.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:01:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:01:09.231 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.232 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.232 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:09.233 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.436808704, 2024-08-30T16:01:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:01:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:01:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:09.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:01:09.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:09.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:01:09.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:01:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:01:09.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:01:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:01:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:01:09.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:01:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:01:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:01:09.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:01:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:01:09.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:01:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:01:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:01:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:01:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:01:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.244 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.435104151, 2024-08-30T16:01:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:01:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.245 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:01:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:01:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:01:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:01:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:01:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:01:09.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:01:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:01:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:01:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:01:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:01:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:01:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:01:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:01:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:01:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:01:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:01:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:01:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:01:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 117.01565034399999, 2024-08-30T16:01:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:01:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.250 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:01:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:01:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:01:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:01:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:01:09.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:01:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:01:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:01:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:01:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:01:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:01:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:01:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:01:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:01:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:01:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:01:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:01:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:01:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:01:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.255 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 116.99876217000001, 2024-08-30T16:01:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:01:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:01:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:01:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:01:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:01:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:01:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:01:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:01:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:01:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:01:09.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:01:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:01:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:01:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:01:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:01:09.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:01:09.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:01:09.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:01:09.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:01:09.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:01:09.260 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.260 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.260 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.261 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.261 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:09.261 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 116.98576210100001, 2024-08-30T16:01:09.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:01:09.261 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.261 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.261 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.261 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.261 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.261 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:01:09.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.262 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:09.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:09.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:01:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:01:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:01:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:01:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:01:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:01:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:01:09.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:01:09.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:01:09.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:01:09.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:01:09.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:01:09.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:01:09.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:01:09.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:01:09.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:01:09.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:01:09.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:01:09.266 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.266 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.266 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.266 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.266 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:09.266 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 116.98326980900001, 2024-08-30T16:01:09.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:01:09.266 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.266 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.266 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.267 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.267 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.267 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:01:09.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.267 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:09.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:09.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:01:09.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:09.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:01:09.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:01:09.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:01:09.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:01:09.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:01:09.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:01:09.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:01:09.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:01:09.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:01:09.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:01:09.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:01:09.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:01:09.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:01:09.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:01:09.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:01:09.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:01:09.271 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.271 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.271 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.271 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.271 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:09.272 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 116.982512361, 2024-08-30T16:01:09.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:01:09.272 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.272 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.272 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.272 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.272 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.272 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:01:09.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.272 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:09.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:09.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:01:09.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:01:09.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:01:09.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:01:09.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:01:09.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:01:09.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:01:09.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:01:09.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:01:09.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:01:09.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:01:09.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:01:09.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:01:09.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:01:09.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:01:09.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:01:09.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:01:09.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:01:09.276 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.276 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.276 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.277 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.277 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:09.277 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 116.981779911, 2024-08-30T16:01:09.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:01:09.277 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.277 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.277 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.277 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.277 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.277 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:01:09.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.278 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:09.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:09.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:01:09.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:09.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:01:09.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:01:09.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:01:09.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:01:09.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:01:09.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:01:09.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:01:09.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:01:09.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:01:09.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:01:09.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:01:09.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:01:09.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:01:09.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:01:09.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:01:09.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:01:09.282 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.282 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.282 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.282 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.282 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:09.282 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 116.979241112, 2024-08-30T16:01:09.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:01:09.282 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.283 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.283 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.283 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.283 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.283 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:01:09.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.283 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:09.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:09.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:01:09.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:09.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:01:09.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:01:09.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:01:09.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:01:09.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:01:09.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:01:09.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:01:09.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:01:09.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:01:09.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:01:09.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:01:09.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:01:09.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:01:09.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:01:09.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:01:09.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:01:09.287 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.287 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.287 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.288 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.288 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:09.288 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 116.97849311100001, 2024-08-30T16:01:09.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:01:09.288 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.288 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.288 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.288 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.288 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.288 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:01:09.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.288 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:09.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:09.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:01:09.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:01:09.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:01:09.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:01:09.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:01:09.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:01:09.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:01:09.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:01:09.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:01:09.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:01:09.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:01:09.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:01:09.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:01:09.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:01:09.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:01:09.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:01:09.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:01:09.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:01:09.293 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.293 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.293 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.293 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.293 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:09.293 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 116.206376639, 2024-08-30T16:01:09.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:01:09.293 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.293 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.294 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.294 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.294 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:01:09.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.294 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:09.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:09.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:01:09.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:01:09.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:01:09.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:01:09.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:01:09.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:01:09.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:01:09.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:01:09.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:01:09.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:01:09.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:01:09.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:01:09.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:01:09.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:01:09.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:01:09.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:01:09.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:01:09.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:01:09.298 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.298 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.298 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.299 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.299 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:09.299 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 116.204319493, 2024-08-30T16:01:09.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:01:09.299 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.299 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.299 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:01:09.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.299 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:09.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:09.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:01:09.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:09.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:01:09.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:01:09.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:01:09.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:01:09.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:01:09.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:01:09.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:01:09.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:01:09.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:01:09.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:01:09.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:01:09.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:01:09.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:01:09.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:01:09.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:01:09.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:01:09.304 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.304 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.304 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.304 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.304 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:09.304 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.941374993, 2024-08-30T16:01:09.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:01:09.304 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.304 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.304 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.304 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.304 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.305 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:01:09.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.305 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:09.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:09.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:01:09.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:01:09.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:01:09.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:01:09.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:01:09.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:01:09.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:01:09.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:01:09.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:01:09.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:01:09.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:01:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:01:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:01:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:01:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:01:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:01:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:01:09.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:01:09.309 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.309 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.309 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.309 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.309 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:09.309 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.94028349, 2024-08-30T16:01:09.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:01:09.309 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.310 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.310 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.310 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.310 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.310 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:01:09.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.310 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.310 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:09.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:09.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:01:09.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:09.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:01:09.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:01:09.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:01:09.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:01:09.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:01:09.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:01:09.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:01:09.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:01:09.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:01:09.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:01:09.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:01:09.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:01:09.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:01:09.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:01:09.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:01:09.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:01:09.314 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.314 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.314 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.314 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.314 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:09.315 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.939533905, 2024-08-30T16:01:09.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:01:09.315 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.315 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.315 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.315 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.315 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.315 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:01:09.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.315 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:09.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:09.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:01:09.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:09.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:01:09.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:01:09.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:01:09.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:01:09.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:01:09.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:01:09.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:01:09.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:01:09.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:01:09.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:01:09.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:01:09.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:01:09.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:01:09.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:01:09.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:01:09.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:01:09.319 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.319 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.319 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.320 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.320 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:09.320 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.905802373, 2024-08-30T16:01:09.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:01:09.320 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.320 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.320 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.320 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.320 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.320 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:01:09.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.321 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:09.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:09.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:01:09.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:01:09.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:01:09.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:01:09.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:01:09.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:01:09.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:01:09.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:01:09.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:01:09.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:01:09.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:01:09.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:01:09.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:01:09.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:01:09.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:01:09.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:01:09.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:01:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:01:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:09.325 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.90527636500001, 2024-08-30T16:01:09.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:01:09.326 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.326 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.326 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.326 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.326 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.326 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:01:09.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.326 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:09.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:09.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:01:09.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:09.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:01:09.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:01:09.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:01:09.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:01:09.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:01:09.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:01:09.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:01:09.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:01:09.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:01:09.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:01:09.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:01:09.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:01:09.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:01:09.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:01:09.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:01:09.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:01:09.330 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.330 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.330 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.331 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:09.331 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:09.331 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.904820802, 2024-08-30T16:01:09.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:01:09.331 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.331 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.331 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.331 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.331 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.331 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:01:09.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.331 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:09.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:09.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:01:09.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:01:09.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:01:09.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:01:09.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:01:09.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:01:09.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:01:09.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:01:09.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:01:09.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:01:09.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:01:09.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:01:09.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:01:09.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:01:09.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:01:09.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:01:09.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:01:09.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:01:09.336 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.336 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.336 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.336 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:01:09.336 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:09.336 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 111.234305486, 2024-08-30T16:01:09.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:01:09.336 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.336 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.336 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:09.337 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:09.337 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:09.337 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:01:09.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.337 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:09.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:09.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:01:09.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:09.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:01:09.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:01:09.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:01:09.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:01:09.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:01:09.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:01:09.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:01:09.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:01:09.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:01:09.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:01:09.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:09.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:01:09.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:01:09.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:01:09.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:01:09.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:01:09.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:01:09.341 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.341 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.341 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.341 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.341 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.341 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:09.343 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:09.343 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:09.341+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:09.443 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:09.442+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:09.464 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-08-30T16:01:09.511 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:09.511 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:09.511 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:09.520 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:09.564 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:09.562+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:09.646 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:09.644+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:09.646 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:09.646 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:09.646 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:09.647 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:09.647 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:09.655 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:09.669 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:10.134 30228== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:09.675 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:10.140 30228== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:09.698 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:10.163 30228== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:09.723 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:42.340 33106== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:09.731 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:42.347 33106== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:09.746 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:09.745+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:09.760 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:42.376 33106== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:09.827 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:09.828 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:09.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:09.828 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:09.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.828 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:01:09.828 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:09.828 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.298058924, 2024-08-30T16:01:09.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:01:09.828 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.828 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.828 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:09.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:09.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:01:09.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:01:09.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:01:09.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:01:09.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:01:09.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:01:09.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:01:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:01:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:01:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:01:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:01:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:01:09.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:01:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:01:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:01:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:01:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.833 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:01:09.833 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:09.833 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 102.179610778, 2024-08-30T16:01:09.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:01:09.833 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:09.833 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:09.833 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:09.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:09.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:09.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:09.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:09.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:09.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:09.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:01:09.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:01:09.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:09.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:01:09.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:01:09.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:09.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:01:09.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:01:09.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:09.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:01:09.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:01:09.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:09.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:01:09.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:01:09.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:09.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:01:09.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:01:09.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:09.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:01:09.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:01:09.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:09.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:09.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:09.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:01:09.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:01:09.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.837 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.837 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:09.837 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:09.837 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:09.838 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:09.882 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:09.880+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:09.948 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:09.946+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:09.965 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:09.966 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:09.966 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:09.966 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:09.974 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:10.049 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:10.047+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:10.074 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:11.019 31218== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:10.135 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:11.080 31218== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:10.143 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:10.143 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:10.143 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:10.143 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:10.143 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:10.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:01:10.150 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:10.148+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:10.153 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:10.250 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:10.249+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:10.340 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:10.340 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:10.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:10.340 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:10.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:10.340 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:10.340 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:10.340 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 101.961600827, 2024-08-30T16:01:10.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:01:10.340 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:10.341 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:10.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:10.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:10.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:10.341 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:01:10.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:10.341 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:10.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:10.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:10.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:10.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:10.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:10.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:10.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:10.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:10.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:10.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:10.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:10.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:10.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:01:10.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:01:10.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:10.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:10.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:10.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:01:10.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:01:10.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:10.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:10.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:10.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:01:10.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:01:10.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:10.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:10.343 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:10.343 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:01:10.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:01:10.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:10.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:10.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:10.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:01:10.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:01:10.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:10.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:10.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:10.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:01:10.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:01:10.344 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:10.344 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:10.344 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:10.344 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:10.345 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:10.345 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:10.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:10.351 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:10.350+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:10.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:01:10.415 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:11.360 31218== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:10.451 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:10.450+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:10.479 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:10.479 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:10.479 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:10.479 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:10.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:10.552 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:10.551+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:10.645 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:10.645 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:10.646 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:10.646 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:10.646 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:10.653 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:10.652+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:10.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:01:10.725 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:01:10.725 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:01:10.754 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:10.753+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:10.809 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:10.809 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:10.818 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:01:10.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:01:10.835 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-08-30T16:01:10.835 INFO:tasks.mon_recovery:stopping all monitors 2024-08-30T16:01:10.835 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-30T16:01:10.835 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:01:10.850 INFO:tasks.ceph.mon.b.smithi094.stderr:2024-08-30T16:01:10.838+0000 a084640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 31216) UID: 0 2024-08-30T16:01:10.851 INFO:tasks.ceph.mon.b.smithi094.stderr:2024-08-30T16:01:10.839+0000 a084640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-08-30T16:01:10.853 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:01:10.854 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:01:10.855 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:10.854+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:10.955 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:10.954+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:10.981 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:01:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:01:10.982 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:10.982 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:10.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:01:11.072 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:01:11.073 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:01:11.073 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:11.071+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:11.119 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:11.119 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:11.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:01:11.158 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:01:11.158 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:01:11.159 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:11.157+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:11.242 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:01:11.243 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:01:11.271 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:11.270+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:11.285 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:11.285 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:11.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:01:11.328 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:01:11.328 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:01:11.372 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:11.371+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:11.411 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:01:11.411 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:01:11.422 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:01:11.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-08-30T16:01:11.456 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:11.456 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:11.459 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:11.458+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:11.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:01:11.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-08-30T16:01:11.560 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:11.559+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:11.640 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:11.641 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:11.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:01:11.691 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:11.690+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:11.762 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:11.761+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:11.816 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:11.817 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:11.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:01:11.866 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:11.865+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:11.965 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:11.964+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:11.999 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:11.999 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:12.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:01:12.093 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:12.092+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:12.177 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:12.178 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:12.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:01:12.199 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:12.198+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:12.270 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:12.269+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:12.357 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:12.357 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:12.367 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:01:12.411 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:12.409+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:12.479 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:12.477+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:12.492 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:12.492 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:12.501 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:01:12.571 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:12.570+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:12.666 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:12.666 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:12.672 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:12.671+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:12.674 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:12.774 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:12.773+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:12.840 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:12.840 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:12.849 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:12.874 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:12.873+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:12.973 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:12.973 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:12.975 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:12.974+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:12.982 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:13.077 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:13.075+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:13.146 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:13.146 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:13.156 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:13.177 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:13.176+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:13.279 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:13.280 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:13.280 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:13.278+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:13.288 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:13.379 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:13.378+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:13.424 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:01:13.454 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:13.455 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:13.463 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:13.508 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:13.506+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:13.581 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:13.580+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:13.591 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:13.591 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:13.600 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:13.673 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:01:13.673 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:01:13.682 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:13.680+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:13.764 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:01:13.765 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:01:13.765 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:13.765 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:13.774 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:13.819 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:13.817+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:13.852 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:01:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:01:13.884 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:13.882+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:13.897 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:13.897 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:13.906 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:13.946 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:01:13.947 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:01:14.049 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:01:14.050 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:14.047+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:14.050 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:01:14.087 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:14.087 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:14.087 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:14.085+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:14.100 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:14.138 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:01:14.138 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:01:14.187 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:14.186+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:14.237 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:01:14.237 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:01:14.288 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:14.287+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:14.330 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:01:14.331 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:01:14.333 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:14.333 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:14.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:14.334 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:14.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:14.334 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:14.334 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:14.334 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 105.955031619, 2024-08-30T16:01:14.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:01:14.334 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:14.335 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:14.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:14.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:14.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:14.335 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:01:14.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:14.336 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:14.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:14.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:14.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:14.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:14.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:14.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:14.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:14.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:14.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:14.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:14.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:14.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:14.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:01:14.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:01:14.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:14.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:14.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:14.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:01:14.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:01:14.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:14.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:14.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:14.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:01:14.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:01:14.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:14.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:14.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:14.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:01:14.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:01:14.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:14.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:14.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:14.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:01:14.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:01:14.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:14.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:14.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:14.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:01:14.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:01:14.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:14.340 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:14.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:14.341 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:14.341 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:14.341 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:14.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:14.426 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:14.425+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:14.491 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:14.489+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:14.519 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:14.519 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:14.520 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:14.520 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:14.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:14.613 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:14.612+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:14.693 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:14.691+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:14.699 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:14.699 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:14.699 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:14.699 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:14.699 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:14.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:01:14.793 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:14.792+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:14.876 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:14.876 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:14.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:14.876 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:01:14.876 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:14.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:01:14.927 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:14.925+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:14.995 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:14.993+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:15.059 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:15.059 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:15.059 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:15.059 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:15.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:01:15.113 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:15.112+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:15.197 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:15.195+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:15.198 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:15.198 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:15.198 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:15.198 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:15.198 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:15.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:01:15.298 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:15.296+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:15.388 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:15.388 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:15.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:15.388 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:15.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.388 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:15.388 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:15.388 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 114.113373814, 2024-08-30T16:01:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:01:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:01:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:01:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:01:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:01:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:01:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:01:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:01:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:01:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:01:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:01:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:15.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:01:15.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:01:15.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:15.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:01:15.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:01:15.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:15.392 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:15.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:15.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.393 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:15.393 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:15.393 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 113.840446453, 2024-08-30T16:01:15.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:01:15.393 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:15.393 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:15.393 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:15.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:15.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:15.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:15.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:01:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:01:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:01:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:01:15.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:15.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:01:15.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:01:15.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:15.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:01:15.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:01:15.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:15.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:01:15.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:01:15.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:15.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:01:15.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:01:15.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:15.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:01:15.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:01:15.397 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:15.397 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:15.397 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:15.397 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:15.397 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:15.397 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:15.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:01:15.399 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:15.397+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:15.500 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:15.498+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:15.527 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:15.528 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:15.528 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:15.528 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:15.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:01:15.600 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:15.599+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:15.667 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:15.667 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:15.667 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:15.668 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:15.668 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:15.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:01:15.720 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:15.717+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:15.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-08-30T16:01:15.802 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:15.800+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:15.847 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:15.847 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:01:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 110.47027350899999, 2024-08-30T16:01:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:01:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:15.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:01:15.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:01:15.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:01:15.851 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:15.852 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:15.852 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:15.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:01:15.903 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:15.901+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:15.987 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:15.987 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:15.987 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:15.987 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:15.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:01:16.003 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:16.002+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:16.104 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:16.103+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:16.163 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:16.163 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:16.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:01:16.215 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:16.214+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:16.306 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:16.304+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:16.323 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:16.323 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:16.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:16.323 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:16.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.323 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:16.324 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:16.324 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 111.872643814, 2024-08-30T16:01:16.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:01:16.324 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:16.324 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:16.324 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:16.324 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:16.324 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:16.324 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:16.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.324 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:16.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.325 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:16.325 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:16.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.325 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:16.325 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:16.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.325 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:16.325 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:01:16.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:01:16.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:16.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:01:16.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:01:16.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:16.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:01:16.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:01:16.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:16.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:01:16.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:01:16.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:16.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:01:16.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:01:16.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:01:16.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:01:16.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:01:16.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:16.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:01:16.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:01:16.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:16.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:01:16.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:01:16.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:01:16.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:01:16.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:01:16.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:16.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:01:16.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:01:16.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:16.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:01:16.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:01:16.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:16.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:01:16.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:01:16.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:16.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:01:16.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:01:16.331 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.331 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:16.331 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.331 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:01:16.331 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:16.331 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 110.75677747899999, 2024-08-30T16:01:16.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:01:16.332 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:16.332 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:16.332 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:16.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:16.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:16.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:16.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:16.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:16.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:01:16.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:16.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:16.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:01:16.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:01:16.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:01:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:01:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:01:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:01:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:16.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:01:16.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:01:16.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:16.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:01:16.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:01:16.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:16.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:01:16.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:01:16.335 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.335 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:16.336 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.336 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:16.336 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:16.336 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 110.697185981, 2024-08-30T16:01:16.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:01:16.336 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:16.336 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:16.336 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:16.336 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:16.336 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:16.336 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:01:16.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.337 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:16.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:16.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:16.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:16.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:16.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:16.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:01:16.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:01:16.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:16.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:01:16.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:01:16.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:16.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:01:16.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:01:16.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:16.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:01:16.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:01:16.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:16.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:01:16.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:01:16.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:16.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:01:16.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:01:16.340 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.340 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:16.340 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.340 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:01:16.340 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:16.340 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 110.670487294, 2024-08-30T16:01:16.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:01:16.340 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:16.340 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:16.341 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:16.341 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:16.341 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:16.341 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:16.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.341 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:16.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:16.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:16.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:16.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:16.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:16.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:01:16.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:16.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:16.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:01:16.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:01:16.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:16.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:01:16.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:01:16.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:16.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:01:16.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:01:16.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:16.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:01:16.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:01:16.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:16.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:01:16.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:01:16.344 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.344 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:16.344 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.344 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:16.345 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:16.345 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 109.802870766, 2024-08-30T16:01:16.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:01:16.345 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:16.345 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:16.345 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:16.345 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:16.345 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:16.345 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:16.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.345 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:16.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:16.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:16.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:16.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:16.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:16.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:01:16.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:16.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:16.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:01:16.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:01:16.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:16.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:01:16.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:01:16.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:16.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:01:16.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:01:16.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:16.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:01:16.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:01:16.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:16.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:01:16.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:01:16.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:16.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:01:16.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:01:16.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:16.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:01:16.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:01:16.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:16.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:01:16.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:01:16.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:16.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:01:16.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:01:16.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:16.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:01:16.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:01:16.351 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.351 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:16.351 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.351 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:01:16.351 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:16.351 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 109.643623687, 2024-08-30T16:01:16.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:01:16.351 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:16.351 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:16.351 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:16.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:16.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:16.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:16.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:16.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:16.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:01:16.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:01:16.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:16.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:01:16.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:01:16.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:16.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:01:16.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:01:16.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:16.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:01:16.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:01:16.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:16.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:01:16.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:01:16.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:16.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:01:16.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:01:16.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:16.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:01:16.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:01:16.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.355 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:16.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.355 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:16.355 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:16.356 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 109.36744962100001, 2024-08-30T16:01:16.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:01:16.356 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:16.356 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:16.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:16.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:16.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:16.356 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:16.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.356 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:16.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:16.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:16.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:16.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:16.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:16.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:01:16.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:16.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:16.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:01:16.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:01:16.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:16.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:01:16.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:01:16.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:16.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:01:16.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:01:16.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:16.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:01:16.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:01:16.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:16.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:01:16.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:01:16.359 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.359 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:16.359 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.360 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:16.360 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:16.360 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 108.709158042, 2024-08-30T16:01:16.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:01:16.360 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:16.360 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:16.360 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:16.360 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:16.360 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:16.360 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:01:16.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.361 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:16.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:16.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:16.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:16.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:16.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:16.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:01:16.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:16.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:16.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:01:16.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:01:16.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:16.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:01:16.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:01:16.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:16.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:01:16.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:01:16.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:16.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:01:16.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:01:16.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:16.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:01:16.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:01:16.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:16.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:01:16.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:01:16.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:16.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:01:16.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:01:16.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:16.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:01:16.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:01:16.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:16.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:01:16.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:01:16.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:16.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:01:16.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:01:16.366 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.366 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:16.366 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.366 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:01:16.366 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:16.366 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 108.61324475799999, 2024-08-30T16:01:16.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:01:16.366 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:16.366 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:16.366 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:01:16.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:16.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:16.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:01:16.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:01:16.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:16.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:01:16.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:01:16.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:16.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:01:16.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:01:16.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:16.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:01:16.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:01:16.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:16.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:01:16.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:01:16.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:16.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:01:16.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:01:16.370 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.370 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:16.370 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.370 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:16.370 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:16.370 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 49.361884246999999, 2024-08-30T16:01:16.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:01:16.371 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:16.371 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:16.371 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:16.371 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:16.371 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:16.371 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:16.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.371 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:16.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:16.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:16.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:16.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:01:16.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:01:16.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:16.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:01:16.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:01:16.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:16.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:01:16.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:01:16.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:16.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:16.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:16.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:01:16.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:01:16.374 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.374 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:16.374 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.374 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:16.374 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:16.374 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:16.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:01:16.376 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:01:16.419 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:16.418+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:16.509 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:16.507+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:16.519 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:16.519 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:16.519 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:16.519 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:16.530 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:01:16.610 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:16.608+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:16.696 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:16.696 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:16.697 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:16.697 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:16.697 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:16.708 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:16.711 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:16.709+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:16.811 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:16.809+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:16.833 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:16.833 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:01:16.833 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:16.842 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:16.911 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:16.910+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:16.937 INFO:tasks.ceph.mon.b:Stopped 2024-08-30T16:01:16.937 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-30T16:01:16.937 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:01:16.953 INFO:tasks.ceph.mon.c.smithi050.stderr:2024-08-30T16:01:16.940+0000 a084640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 33100) UID: 0 2024-08-30T16:01:16.954 INFO:tasks.ceph.mon.c.smithi050.stderr:2024-08-30T16:01:16.942+0000 a084640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-08-30T16:01:17.006 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:17.006 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:17.006 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:17.007 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:17.012 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:17.010+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:17.016 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:17.113 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:17.111+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:17.182 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:17.182 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:17.182 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:17.191 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:17.235 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:17.233+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:17.315 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:17.313+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:17.370 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:17.371 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:17.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:17.371 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:17.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.371 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:17.371 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:17.371 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 126.508708391, 2024-08-30T16:01:17.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:01:17.371 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.371 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.371 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.372 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.372 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.372 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:01:17.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.372 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:17.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:17.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:01:17.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:01:17.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:01:17.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:01:17.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:01:17.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:01:17.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:01:17.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:01:17.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:01:17.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:01:17.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:01:17.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:01:17.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:01:17.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:01:17.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:01:17.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:01:17.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:01:17.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:01:17.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:01:17.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:01:17.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:01:17.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:01:17.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:01:17.377 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.377 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.378 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.378 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:17.378 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:17.378 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 125.592292371, 2024-08-30T16:01:17.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:01:17.378 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.378 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.378 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.378 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.379 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.379 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:01:17.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.379 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:17.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:17.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:01:17.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:17.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.380 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:01:17.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:01:17.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:01:17.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:01:17.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:01:17.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:01:17.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:01:17.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:01:17.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:01:17.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:01:17.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:01:17.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:01:17.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:01:17.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:01:17.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:01:17.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:01:17.383 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.383 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.383 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.384 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:17.384 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:17.384 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 125.590587818, 2024-08-30T16:01:17.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:01:17.384 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.384 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.384 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.384 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.384 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.384 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:01:17.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.384 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:17.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.385 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.385 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:17.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:01:17.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:01:17.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:01:17.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:01:17.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:01:17.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:01:17.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:01:17.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:01:17.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:01:17.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:01:17.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:01:17.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:01:17.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:01:17.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:01:17.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:01:17.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:01:17.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:01:17.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:01:17.389 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.389 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.389 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.389 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.390 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:17.390 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 125.17113401100001, 2024-08-30T16:01:17.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:01:17.390 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.390 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.390 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.390 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.390 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.390 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:01:17.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.390 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:17.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:17.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:01:17.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:01:17.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:01:17.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:01:17.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:01:17.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:01:17.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:01:17.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:01:17.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:01:17.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:01:17.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:01:17.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:01:17.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:01:17.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:01:17.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:01:17.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:01:17.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:01:17.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:01:17.395 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.395 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.395 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.395 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.395 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:17.395 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 125.154245837, 2024-08-30T16:01:17.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:01:17.395 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.396 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.396 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.396 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.396 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.396 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:01:17.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.396 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:17.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:17.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:01:17.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:01:17.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:01:17.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:01:17.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:01:17.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:01:17.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:01:17.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:01:17.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:01:17.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:01:17.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:01:17.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:01:17.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:01:17.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:01:17.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:01:17.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:01:17.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:01:17.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:01:17.400 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.400 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.400 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.401 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.401 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:17.401 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 125.141245768, 2024-08-30T16:01:17.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:01:17.401 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.401 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.401 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.401 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.401 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.401 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:01:17.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.402 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:17.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:17.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:01:17.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:01:17.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:01:17.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:01:17.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:01:17.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:01:17.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:01:17.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:01:17.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:01:17.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:01:17.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:01:17.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:01:17.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:01:17.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:01:17.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:01:17.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:01:17.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:01:17.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:01:17.406 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.406 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.406 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.406 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.406 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:17.407 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 125.13875347600001, 2024-08-30T16:01:17.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:01:17.407 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.407 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.407 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.407 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.407 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.407 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:01:17.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.407 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:17.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:17.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:01:17.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:17.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:01:17.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:01:17.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:01:17.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:01:17.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:01:17.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:01:17.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:01:17.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:01:17.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:01:17.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:01:17.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:01:17.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:01:17.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:01:17.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:01:17.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:01:17.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:01:17.411 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.412 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.412 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.412 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:17.412 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 125.137996028, 2024-08-30T16:01:17.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:01:17.412 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.412 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.413 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.413 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:01:17.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.413 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:17.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:17.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:01:17.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:01:17.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:01:17.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:01:17.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:01:17.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:01:17.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:01:17.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:01:17.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:01:17.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:01:17.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:01:17.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:01:17.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:01:17.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:01:17.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:01:17.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:01:17.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:01:17.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:01:17.417 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.417 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.417 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.418 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.418 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:17.418 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 125.137263578, 2024-08-30T16:01:17.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:01:17.418 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.418 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.418 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:01:17.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.418 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:17.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:17.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:01:17.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:17.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:01:17.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:01:17.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:01:17.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:01:17.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:01:17.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:01:17.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:01:17.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:01:17.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:01:17.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:01:17.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:01:17.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:01:17.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:01:17.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:01:17.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:01:17.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:01:17.423 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.423 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.423 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.423 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.423 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:17.423 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 125.134724779, 2024-08-30T16:01:17.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:01:17.423 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.424 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.424 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.424 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.424 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.424 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:01:17.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.424 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:17.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:17.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:01:17.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:17.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:01:17.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:01:17.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:01:17.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:01:17.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:01:17.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:01:17.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:01:17.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:01:17.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:01:17.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:01:17.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:01:17.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:01:17.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:01:17.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:01:17.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:01:17.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:01:17.429 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.429 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.429 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.429 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.429 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:17.429 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 125.133976778, 2024-08-30T16:01:17.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:01:17.429 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.429 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.430 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:01:17.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.430 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:17.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:17.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:01:17.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:01:17.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:01:17.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:01:17.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:01:17.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:01:17.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:01:17.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:01:17.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:01:17.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:01:17.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:01:17.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:01:17.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:01:17.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:01:17.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:01:17.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:01:17.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:01:17.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:01:17.434 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.434 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.434 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.435 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.435 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:17.435 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 124.361860306, 2024-08-30T16:01:17.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:01:17.435 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.435 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.435 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.435 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.435 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.435 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:01:17.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.436 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:17.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:17.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:01:17.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:01:17.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:01:17.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:01:17.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:01:17.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:01:17.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:01:17.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:01:17.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:01:17.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:01:17.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:01:17.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:01:17.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:01:17.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:01:17.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:01:17.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:01:17.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:01:17.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:01:17.440 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.440 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.440 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.440 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.440 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:17.440 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 124.35980316, 2024-08-30T16:01:17.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:01:17.441 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.441 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.441 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.441 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.441 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.441 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:01:17.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.441 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:17.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:17.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:01:17.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:17.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:01:17.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:01:17.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:01:17.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:01:17.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:01:17.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:01:17.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:01:17.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:01:17.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:01:17.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:01:17.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:01:17.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:01:17.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:01:17.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:01:17.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:01:17.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:01:17.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.445 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.446 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.446 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:17.446 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 124.09685866, 2024-08-30T16:01:17.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:01:17.446 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.446 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.446 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.446 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.446 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.446 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:01:17.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.447 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:17.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:17.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:01:17.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:01:17.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:01:17.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:01:17.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:01:17.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:01:17.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:01:17.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:01:17.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:01:17.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:01:17.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:01:17.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:01:17.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:01:17.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:01:17.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:01:17.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:01:17.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:01:17.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:01:17.451 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.451 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.452 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.452 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.452 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:17.452 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 124.095767157, 2024-08-30T16:01:17.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:01:17.452 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.452 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.452 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.452 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.452 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.453 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:01:17.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.453 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:17.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:17.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:01:17.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:17.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:01:17.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:01:17.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:01:17.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:01:17.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:01:17.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:01:17.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:01:17.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:01:17.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:01:17.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:01:17.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:01:17.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:01:17.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:01:17.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:01:17.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:01:17.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:01:17.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.457 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.458 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.458 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:17.458 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 124.095017572, 2024-08-30T16:01:17.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:01:17.458 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.458 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.459 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:01:17.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.459 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:17.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:17.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:01:17.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:17.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:01:17.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:01:17.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:01:17.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:01:17.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:01:17.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:01:17.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:01:17.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:01:17.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:01:17.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:01:17.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:01:17.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:01:17.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:01:17.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:01:17.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:01:17.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:01:17.463 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.463 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.463 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.464 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.464 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:17.464 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 124.06128604, 2024-08-30T16:01:17.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:01:17.464 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.464 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.464 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.464 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.464 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.464 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:01:17.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.465 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:17.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:17.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:01:17.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:01:17.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:01:17.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:01:17.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:01:17.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:01:17.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:01:17.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:01:17.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:01:17.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:01:17.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:01:17.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:01:17.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:01:17.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:01:17.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:01:17.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:01:17.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:01:17.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:01:17.469 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.469 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.469 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.469 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.469 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:17.470 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 124.060760032, 2024-08-30T16:01:17.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:01:17.470 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.470 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.470 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.470 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.470 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.470 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:01:17.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.470 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:17.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:17.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:01:17.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:17.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:01:17.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:01:17.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:01:17.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:01:17.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:01:17.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:01:17.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:01:17.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:01:17.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:01:17.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:01:17.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:01:17.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:01:17.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:01:17.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:01:17.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:01:17.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:01:17.475 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.475 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.475 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.475 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:17.475 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:17.475 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 124.060304469, 2024-08-30T16:01:17.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:01:17.475 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.475 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.476 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.476 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.476 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:01:17.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.476 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:17.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:17.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:01:17.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:01:17.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:01:17.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:01:17.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:01:17.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:01:17.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:01:17.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:01:17.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:01:17.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:01:17.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:01:17.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:01:17.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:01:17.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:01:17.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:01:17.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:01:17.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:01:17.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:01:17.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.482 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.482 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:01:17.482 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:17.482 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 119.389789153, 2024-08-30T16:01:17.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:01:17.482 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.482 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.482 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:17.482 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:17.483 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:17.483 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:01:17.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.483 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:17.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:17.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:01:17.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:17.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:01:17.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:01:17.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:01:17.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:01:17.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:01:17.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:01:17.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:01:17.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:01:17.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:01:17.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:01:17.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:17.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:01:17.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:01:17.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:01:17.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:01:17.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:01:17.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:01:17.491 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.491 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.491 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.491 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.491 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.491 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:17.493 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:17.493 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:17.491+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:17.593 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:17.592+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:17.664 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:17.664 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:17.664 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:17.664 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:17.674 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:17.695 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:17.693+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:17.796 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:17.795+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:17.798 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:17.798 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:17.798 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:17.798 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:17.798 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:17.807 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:17.897 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:17.896+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:17.939 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:17.939 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:17.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:17.940 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:17.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.940 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:01:17.940 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:17.940 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 111.409805401, 2024-08-30T16:01:17.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:01:17.940 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.940 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.940 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:17.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:17.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:01:17.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:01:17.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:01:17.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:01:17.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:01:17.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:01:17.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:01:17.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:01:17.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:01:17.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:01:17.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:01:17.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:01:17.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:01:17.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:01:17.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:01:17.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:01:17.944 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.944 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.944 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.945 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:01:17.945 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:17.945 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.29135725499999, 2024-08-30T16:01:17.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:01:17.945 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:17.945 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:17.945 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:17.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:17.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:17.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:17.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:17.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:17.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:17.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:01:17.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:01:17.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:17.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:01:17.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:01:17.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:17.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:01:17.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:01:17.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:17.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:01:17.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:01:17.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:17.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:01:17.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:01:17.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:17.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:01:17.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:01:17.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:17.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:01:17.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:01:17.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:17.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:17.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:17.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:01:17.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:01:17.949 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.949 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.949 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.949 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:17.949 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:17.949 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:17.950 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:17.998 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:17.997+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:18.099 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:18.098+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:18.115 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:18.115 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:18.115 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:18.115 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:18.124 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:18.200 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:18.199+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:18.251 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:18.252 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:18.252 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:18.252 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:18.252 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:18.261 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:18.301 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:18.299+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:18.402 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:18.401+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:18.433 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:18.433 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:18.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:18.434 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:18.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:18.434 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:18.434 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:18.434 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 110.05552903900001, 2024-08-30T16:01:18.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:01:18.434 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:18.434 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:18.434 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:18.434 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:18.434 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:18.434 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:01:18.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:18.435 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:18.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:18.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:18.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:18.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:18.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:18.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:18.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:18.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:18.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:18.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:18.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:18.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:18.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:01:18.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:01:18.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:18.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:18.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:18.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:01:18.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:01:18.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:18.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:18.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:18.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:01:18.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:01:18.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:18.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:18.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:18.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:01:18.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:01:18.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:18.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:18.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:18.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:01:18.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:01:18.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:18.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:18.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:18.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:01:18.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:01:18.438 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:18.438 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:18.438 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:18.438 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:18.438 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:18.438 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:18.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:18.503 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:18.501+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:18.577 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:18.577 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:18.577 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:18.577 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:18.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:18.603 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:18.601+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:18.704 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:18.703+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:18.712 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:18.712 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:18.712 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:18.712 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:18.713 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:18.722 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:01:18.805 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:18.804+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:18.894 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:18.894 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:18.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:18.894 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:01:18.894 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:18.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:01:18.907 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:18.905+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:19.008 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:19.006+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:19.033 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:19.034 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:19.034 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:19.034 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:19.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:01:19.109 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:19.107+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:19.170 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:19.171 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:19.171 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:19.171 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:19.171 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:19.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:01:19.221 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:19.220+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:19.311 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:19.310+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:19.355 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:19.355 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:19.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:19.355 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:19.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.356 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:19.356 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:19.356 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 118.07815755599999, 2024-08-30T16:01:19.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:01:19.356 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:19.356 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:19.357 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:19.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:19.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:19.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:19.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:19.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:19.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:19.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:19.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:01:19.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:01:19.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:19.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:01:19.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:01:19.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:19.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:01:19.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:01:19.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:19.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:01:19.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:01:19.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:19.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:01:19.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:01:19.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:19.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:01:19.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:01:19.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:19.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:01:19.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:01:19.363 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:19.363 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:19.363 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:19.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.363 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:19.364 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:19.364 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 117.80523019500001, 2024-08-30T16:01:19.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:01:19.364 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:19.364 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:19.364 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:19.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:19.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:19.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:19.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:19.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:19.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:19.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:19.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:01:19.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:01:19.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:19.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:01:19.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:01:19.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:19.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:01:19.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:01:19.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:19.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:01:19.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:01:19.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:19.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:01:19.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:01:19.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:19.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:01:19.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:01:19.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:19.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:01:19.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:01:19.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:19.369 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:19.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:19.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:19.369 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:19.369 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:19.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:01:19.411 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:19.410+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:19.513 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:19.511+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:19.539 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:19.540 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:19.540 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:19.540 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:19.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:01:19.634 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:19.633+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:19.714 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:19.713+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:19.717 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:19.717 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:19.718 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:19.718 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:19.718 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:19.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:01:19.816 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:19.814+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:19.898 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:01:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 114.520512117, 2024-08-30T16:01:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:01:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:19.898 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:19.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:01:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:01:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:01:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:01:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:01:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:01:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:01:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:01:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:01:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:01:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:01:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:01:19.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:01:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:01:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:01:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:01:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:19.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:19.903 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:19.903 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:19.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:01:19.952 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:19.950+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:20.018 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:20.016+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:20.037 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:20.037 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:20.037 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:20.037 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:20.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:01:20.131 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:20.130+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:20.216 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:20.216 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:20.216 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:20.216 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:20.216 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:20.218 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:20.217+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:20.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:01:20.319 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:20.318+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:20.418 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:20.418 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:20.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:20.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:20.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.418 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:20.418 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:20.418 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 115.967155874, 2024-08-30T16:01:20.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:01:20.419 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:20.419 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:20.419 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:20.419 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:20.419 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:20.419 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:20.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.419 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:20.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:20.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:20.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:20.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:20.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:20.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:01:20.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:01:20.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:20.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:01:20.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:01:20.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:20.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:01:20.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:01:20.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:20.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:01:20.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:01:20.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:20.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:01:20.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:01:20.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:01:20.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:01:20.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:01:20.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:20.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:01:20.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:01:20.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:20.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:01:20.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:01:20.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:01:20.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:01:20.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:01:20.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:20.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:01:20.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:01:20.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:20.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:01:20.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:01:20.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:20.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:01:20.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:01:20.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:20.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:01:20.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:01:20.426 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.426 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:20.426 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.426 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:01:20.426 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:20.426 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 114.85128953900001, 2024-08-30T16:01:20.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:01:20.426 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:20.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:01:20.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:20.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:20.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:01:20.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:01:20.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:20.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:01:20.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:01:20.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:20.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:01:20.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:01:20.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:20.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:01:20.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:01:20.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:20.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:01:20.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:01:20.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:20.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:01:20.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:01:20.430 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.430 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:20.430 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.430 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:20.431 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:20.431 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 114.791698041, 2024-08-30T16:01:20.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:01:20.431 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:20.431 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:20.431 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:20.431 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:20.431 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:20.431 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:01:20.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.431 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:20.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:20.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:20.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:20.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:20.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:20.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:01:20.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:01:20.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:01:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:01:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:01:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:01:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:20.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:01:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:01:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:01:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:01:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:01:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:01:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.434 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:01:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 114.764999354, 2024-08-30T16:01:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:01:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:20.435 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:01:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:01:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:01:20.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:01:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:01:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:01:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:01:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:20.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 113.897382826, 2024-08-30T16:01:20.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:01:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:01:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:01:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:01:20.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:01:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:01:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:01:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:01:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:01:20.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:01:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:01:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:01:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:01:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:01:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:01:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:01:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:01:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:01:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:01:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:01:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:01:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:01:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:01:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 113.738135747, 2024-08-30T16:01:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:01:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:01:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:01:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:01:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:01:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:01:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:01:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:20.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:01:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:01:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:01:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:01:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:01:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:01:20.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:01:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:01:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 113.46196168100001, 2024-08-30T16:01:20.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:01:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:01:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:01:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:01:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:01:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:01:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:01:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:01:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:01:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:01:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:01:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:01:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 112.803670102, 2024-08-30T16:01:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:01:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:01:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:01:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:01:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:01:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:01:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:01:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:01:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:01:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:01:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:01:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:01:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:01:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:01:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:01:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:01:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:01:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:01:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:01:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:01:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:01:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:01:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:01:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:01:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 112.70775681800001, 2024-08-30T16:01:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:01:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:01:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:01:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:01:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:01:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:01:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:01:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:01:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:01:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:01:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:01:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:01:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:01:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:01:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.465 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 53.456396306999999, 2024-08-30T16:01:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:01:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:01:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:01:20.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:01:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:01:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:01:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:01:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:01:20.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:01:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:01:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:01:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:01:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:01:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:20.469 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.470 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:20.470 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:20.470 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:20.471 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:01:20.471 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:20.470+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:20.573 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:20.571+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:20.611 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:20.611 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:20.612 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:20.620 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:01:20.705 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:20.703+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:20.773 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:20.772+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:20.786 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:20.787 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:20.796 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:20.880 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:20.879+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:20.963 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:20.963 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:20.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:20.963 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:01:20.963 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:20.973 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:21.014 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:21.012+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:21.076 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:21.074+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:21.136 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:21.136 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:21.136 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:21.136 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:21.145 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:21.176 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:21.175+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:21.268 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:21.268 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:21.269 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:21.269 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:21.269 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:21.277 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:21.278 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:21.276+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:21.379 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:21.377+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:21.480 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:21.478+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:21.489 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 130.626694692, 2024-08-30T16:01:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:01:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.490 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:01:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:01:21.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:01:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:01:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:01:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:01:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:01:21.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:01:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:01:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:01:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:01:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:01:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:01:21.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:01:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:01:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:01:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:01:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:01:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:01:21.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:01:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:01:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:01:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:01:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:01:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.498 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 129.71027867199999, 2024-08-30T16:01:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:01:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.499 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:01:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:01:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:01:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:01:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:01:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:01:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:01:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:01:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:01:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:01:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:01:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:01:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:01:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:01:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:01:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:01:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:01:21.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:01:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 129.70857411899999, 2024-08-30T16:01:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:01:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.505 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:01:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:21.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:01:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:01:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:01:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:01:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:01:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:01:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:01:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:01:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:01:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:01:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:01:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:01:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:01:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:01:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:01:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:01:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:01:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:01:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.510 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 129.28912031199999, 2024-08-30T16:01:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:01:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:01:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:01:21.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:01:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:01:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:01:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:01:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:01:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:01:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:01:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:01:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:01:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:01:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:01:21.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:01:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:01:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:01:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:01:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:01:21.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:01:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 129.27223213799999, 2024-08-30T16:01:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:01:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.516 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:01:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:01:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:01:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:01:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:01:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:01:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:01:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:01:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:01:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:01:21.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:01:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:01:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:01:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:01:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:01:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:01:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:01:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:01:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:01:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.521 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 129.25923206900001, 2024-08-30T16:01:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:01:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:01:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:01:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:01:21.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:01:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:01:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:01:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:01:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:01:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:01:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:01:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:01:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:01:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:01:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:01:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:01:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:01:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:01:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:01:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:01:21.526 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 129.25673977700001, 2024-08-30T16:01:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:01:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:01:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:01:21.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:01:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:01:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:01:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:01:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:01:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:01:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:01:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:01:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:01:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:01:21.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:01:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:01:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 129.25598232900001, 2024-08-30T16:01:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:01:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:01:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.533 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:01:21.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:01:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:01:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:01:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:01:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:01:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:01:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:01:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:01:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:01:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:01:21.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:01:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:01:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:01:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:01:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:01:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:01:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:01:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 129.25524987899999, 2024-08-30T16:01:21.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:01:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:01:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:21.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:01:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:21.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:01:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:01:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:01:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:01:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:01:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:01:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:01:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:01:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:01:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:01:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:01:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:01:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:01:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:01:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:01:21.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:01:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 129.25271108000001, 2024-08-30T16:01:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:01:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.545 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:01:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:01:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:01:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:01:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:01:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:01:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:01:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:01:21.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:01:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:01:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:01:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:01:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:01:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:01:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:01:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:01:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:01:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:01:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.550 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 129.25196307900001, 2024-08-30T16:01:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:01:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.551 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:01:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:01:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:01:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:01:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:01:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:01:21.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:01:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:01:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:01:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:01:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:01:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:01:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:01:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:01:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:01:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:01:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:01:21.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:01:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:01:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.47984660700001, 2024-08-30T16:01:21.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:01:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:01:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:01:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:01:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:01:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:01:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:01:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:01:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:01:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:01:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:01:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:01:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:01:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:01:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:01:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:01:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:01:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:01:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:01:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:01:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.47778946099999, 2024-08-30T16:01:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:01:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:01:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:01:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:01:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:01:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:01:21.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:01:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:01:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:01:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:01:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:01:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:01:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:01:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:01:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:01:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:01:21.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:01:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:01:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:01:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.567 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.21484496100001, 2024-08-30T16:01:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:01:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:01:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:01:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:01:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:01:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:01:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:01:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:01:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:01:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:01:21.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:01:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:01:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:01:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:01:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:01:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:01:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:01:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:01:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:01:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:01:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.21375345800001, 2024-08-30T16:01:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:01:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.573 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:01:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:01:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:01:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:01:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:01:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:01:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:01:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:01:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:01:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:01:21.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:01:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:01:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:01:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:01:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:01:21.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:01:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:01:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:01:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.21300387299999, 2024-08-30T16:01:21.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:01:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:01:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:01:21.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:01:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:01:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:01:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:01:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:01:21.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:01:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:01:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:01:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:01:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:01:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:01:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:01:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:01:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:01:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.583 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.179272341, 2024-08-30T16:01:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:01:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:01:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:01:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:01:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:01:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:01:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:01:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:01:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:01:21.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:01:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:01:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:01:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:01:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:01:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:01:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:01:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:01:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:01:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:01:21.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:01:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.17874633299999, 2024-08-30T16:01:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:01:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.589 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:01:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:21.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:01:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:01:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:01:21.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:01:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:01:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:01:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:01:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:01:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:01:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:01:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:01:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:01:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:01:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:01:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:01:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:01:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:01:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 128.17829076999999, 2024-08-30T16:01:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:01:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.595 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:01:21.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.596 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:21.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:01:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:01:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:01:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:01:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:01:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:01:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:01:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:01:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:01:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:01:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:01:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:01:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:01:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:01:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:01:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:01:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:01:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:01:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:01:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:21.601 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.507775454, 2024-08-30T16:01:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:01:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:01:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:01:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:01:21.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:01:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:01:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:01:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:01:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:01:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:01:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:01:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:01:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:01:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:01:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:01:21.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:01:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:01:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:01:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:01:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:21.606 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:21.607 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:21.608 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:21.608 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:21.606+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:21.708 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:21.707+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:21.781 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:21.781 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:21.781 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:21.788 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:21.810 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:21.809+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:21.910 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-08-30T16:01:21.910 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-08-30T16:01:21.910 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:21.909+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:21.954 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:21.955 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:21.964 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:22.012 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:22.011+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:22.113 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:22.111+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:22.136 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:22.136 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:22.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:22.137 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:22.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.137 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:01:22.137 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:22.137 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.606827671, 2024-08-30T16:01:22.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:01:22.137 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:22.137 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:22.137 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:22.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:22.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:22.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:22.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:22.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:22.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:22.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:22.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:01:22.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:01:22.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:22.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:01:22.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:01:22.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:22.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:01:22.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:01:22.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:22.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:01:22.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:01:22.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:22.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:01:22.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:01:22.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:22.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:01:22.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:01:22.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:22.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:01:22.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:01:22.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:22.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:01:22.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:01:22.141 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:22.141 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:22.141 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:22.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.141 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:01:22.142 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:22.142 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 114.488379525, 2024-08-30T16:01:22.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:01:22.142 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:22.142 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:22.142 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:22.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:22.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:22.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:22.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:22.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:22.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:22.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:22.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:01:22.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:01:22.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:22.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:01:22.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:01:22.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:22.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:01:22.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:01:22.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:22.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:01:22.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:01:22.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:22.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:01:22.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:01:22.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:22.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:01:22.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:01:22.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:22.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:01:22.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:01:22.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:22.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:01:22.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:01:22.146 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:22.146 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:22.146 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:22.146 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:22.146 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:22.146 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:22.147 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:22.232 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:22.230+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:22.311 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:22.312 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:22.312 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:22.312 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:22.314 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:22.313+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:22.321 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:22.415 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:22.414+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:22.493 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:22.493 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:22.493 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:22.493 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:22.493 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:22.502 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:22.546 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:22.544+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:22.616 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:22.615+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:22.630 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:22.630 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 114.25243546199999, 2024-08-30T16:01:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:01:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:01:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:22.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:01:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:01:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:22.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:01:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:01:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:01:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:01:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:01:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:01:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:01:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:01:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:01:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:01:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:22.634 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:22.634 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:22.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:22.718 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:22.716+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:22.783 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:22.784 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:22.784 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:22.784 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:22.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:22.819 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:22.817+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:22.920 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:22.919+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:22.962 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:22.962 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:22.963 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:22.963 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:22.963 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:22.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:01:23.020 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:23.019+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:23.039 INFO:tasks.ceph.mon.c:Stopped 2024-08-30T16:01:23.039 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-30T16:01:23.039 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:01:23.080 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T16:01:23.062+0000 a084640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 30220) UID: 0 2024-08-30T16:01:23.081 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T16:01:23.064+0000 a084640 -1 mon.a@0(probing) e1 *** Got Signal Terminated *** 2024-08-30T16:01:23.121 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:23.120+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:23.155 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:23.156 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:23.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:23.156 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:01:23.156 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:23.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:01:23.249 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:23.248+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:23.324 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:23.323+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:23.337 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:23.337 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:23.337 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:23.337 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:23.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:01:23.425 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:23.423+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:23.516 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:23.516 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:23.517 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:23.517 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:23.525 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:23.524+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:23.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:01:23.626 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:23.625+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:23.661 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:23.662 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:23.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:23.662 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:23.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.662 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:23.662 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:23.662 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 122.387005399, 2024-08-30T16:01:23.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:01:23.662 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:23.662 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:23.663 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:23.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:23.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:23.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:23.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:23.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:23.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:23.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:23.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:01:23.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:01:23.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:23.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:01:23.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:01:23.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:23.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:01:23.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:01:23.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:23.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:01:23.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:01:23.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:23.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:01:23.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:01:23.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:23.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:01:23.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:01:23.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:23.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:01:23.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:01:23.666 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:23.666 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:23.666 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:23.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.666 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:23.667 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:23.667 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 122.114078038, 2024-08-30T16:01:23.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:01:23.667 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:23.667 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:23.667 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:23.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:23.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:23.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:23.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:23.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:23.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:23.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:23.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:01:23.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:01:23.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:23.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:01:23.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:01:23.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:23.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:01:23.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:01:23.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:23.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:01:23.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:01:23.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:23.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:01:23.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:01:23.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:23.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:01:23.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:01:23.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:23.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:23.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:23.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:01:23.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:01:23.670 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:23.670 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:23.671 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:23.671 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:23.671 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:23.671 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:23.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:01:23.756 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:23.755+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:23.828 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:23.827+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:23.841 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:23.841 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:23.841 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:23.841 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:23.850 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:01:23.934 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:23.933+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:24.017 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:24.017 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:24.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:01:24.030 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:24.029+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:24.130 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:24.129+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:24.199 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:24.199 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:24.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:24.199 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:24.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.199 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:01:24.200 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:24.200 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 118.821943906, 2024-08-30T16:01:24.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:01:24.200 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:24.200 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:24.200 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:24.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:24.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:24.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.201 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:24.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:24.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.201 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:24.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:01:24.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:01:24.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:24.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:01:24.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:01:24.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:24.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:01:24.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:01:24.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:24.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:01:24.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:01:24.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:24.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:01:24.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:01:24.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:24.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:01:24.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:01:24.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:24.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:01:24.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:01:24.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:24.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:01:24.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:01:24.205 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.205 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:24.205 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.205 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.205 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:24.205 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:24.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:01:24.231 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:24.229+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:24.331 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:24.330+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:24.379 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:24.379 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:24.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:01:24.432 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:24.430+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:24.533 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:24.531+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:24.555 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:24.555 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:24.555 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:24.555 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:24.556 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:24.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:01:24.633 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:24.632+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:24.715 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:24.715 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:24.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:24.715 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:24.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.716 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:24.716 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:24.716 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 120.26400300900001, 2024-08-30T16:01:24.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:01:24.716 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:24.716 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:24.716 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:24.716 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:24.716 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:24.716 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:24.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.716 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:24.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:24.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:24.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:24.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:24.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:24.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:01:24.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:01:24.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:24.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:01:24.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:01:24.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:24.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:01:24.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:01:24.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:24.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:01:24.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:01:24.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:24.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:01:24.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:01:24.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:01:24.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:01:24.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:01:24.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:24.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:01:24.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:01:24.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:24.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:01:24.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:01:24.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:01:24.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:01:24.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:01:24.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:24.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:01:24.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:01:24.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:24.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:01:24.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:01:24.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:24.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:01:24.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:01:24.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:24.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:01:24.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:01:24.722 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.722 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:24.722 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.723 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:01:24.723 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:24.723 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.148136674, 2024-08-30T16:01:24.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:01:24.723 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:24.723 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:24.723 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:24.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:24.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:24.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:24.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:24.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:24.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:01:24.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:24.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:01:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:01:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:01:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:01:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:01:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:01:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:24.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:01:24.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:01:24.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:24.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:01:24.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:01:24.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:24.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:01:24.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:01:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.088545176, 2024-08-30T16:01:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:01:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:24.728 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:01:24.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.728 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:24.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:24.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:24.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:24.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:24.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:24.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:01:24.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:01:24.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:24.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:01:24.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:01:24.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:24.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:01:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:01:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:01:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:01:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:01:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:01:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:01:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:01:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:01:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 119.061846489, 2024-08-30T16:01:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:01:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:01:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:01:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:01:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:01:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:01:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:01:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:01:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:01:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:01:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:01:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:01:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:24.735 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:24.736 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 118.194229961, 2024-08-30T16:01:24.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:01:24.736 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:24.736 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:24.736 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:24.736 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:24.736 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:24.736 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:24.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.736 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:24.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:24.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:01:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:24.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:01:24.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:01:24.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:24.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:01:24.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:01:24.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:24.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:01:24.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:01:24.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:24.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:01:24.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:01:24.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:24.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:01:24.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:01:24.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:24.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:01:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:01:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:01:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:01:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:01:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:01:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:24.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:01:24.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:01:24.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:24.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:01:24.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:01:24.741 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.741 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:24.741 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.741 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:01:24.742 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:24.742 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 118.03498288199999, 2024-08-30T16:01:24.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:01:24.742 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:24.742 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:24.742 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:24.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:24.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:24.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:24.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:24.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:24.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:01:24.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:01:24.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:24.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:01:24.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:01:24.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:24.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:01:24.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:01:24.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.744 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:24.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:01:24.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:01:24.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.744 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:24.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:01:24.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:01:24.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:24.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:01:24.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:01:24.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:24.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:01:24.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:01:24.745 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.745 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:24.745 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 117.758808816, 2024-08-30T16:01:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:01:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:24.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:24.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:24.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:24.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:24.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:24.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:01:24.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:24.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:24.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:01:24.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:01:24.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:24.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:01:24.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:01:24.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.749 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:24.749 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:01:24.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:01:24.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.749 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:24.749 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:01:24.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:01:24.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.749 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:24.749 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:01:24.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:01:24.749 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.750 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:24.750 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.750 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.750 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:24.750 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:24.750 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 117.10051723700001, 2024-08-30T16:01:24.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:01:24.750 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:24.750 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:24.750 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:24.750 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:24.750 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:24.751 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:01:24.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.751 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:24.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:24.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:24.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:24.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:24.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:24.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:01:24.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:24.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:24.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:01:24.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:01:24.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:24.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:01:24.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:01:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:01:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:01:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:01:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:01:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:24.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:01:24.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:01:24.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:24.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:01:24.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:01:24.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:24.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:01:24.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:01:24.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:24.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:01:24.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:01:24.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:24.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:01:24.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:01:24.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:24.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:01:24.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:01:24.755 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.755 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:24.756 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.756 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:01:24.756 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:24.756 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 117.004603953, 2024-08-30T16:01:24.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:01:24.756 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:24.756 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:24.756 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:24.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:24.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:24.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:24.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:24.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:24.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:01:24.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:24.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:24.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:01:24.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:01:24.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:24.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:01:24.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:01:24.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:24.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:01:24.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:01:24.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:01:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:01:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:01:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:01:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:01:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:01:24.759 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.760 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:24.760 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.760 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:24.760 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:24.760 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.753243441999999, 2024-08-30T16:01:24.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:01:24.760 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:24.760 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:24.760 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:24.760 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:24.760 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:24.761 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:24.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.761 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:24.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:24.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:24.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:24.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:24.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:24.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:24.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:01:24.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:01:24.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:24.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:01:24.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:01:24.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:24.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:01:24.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:01:24.763 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:24.763 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:01:24.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:01:24.763 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:24.763 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:01:24.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:01:24.763 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:24.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:24.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:24.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:01:24.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:01:24.764 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.764 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:24.764 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.764 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:24.764 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:24.764 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:24.766 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:01:24.766 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:24.764+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:24.867 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:24.866+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:24.906 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:24.906 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:24.906 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:24.916 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:01:24.968 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:24.967+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:25.069 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:25.068+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:25.085 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:25.085 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:25.085 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:25.085 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:25.085 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:25.095 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:25.170 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:25.169+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:25.273 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:25.274 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:25.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:25.274 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:01:25.274 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:25.274 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:25.272+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:25.283 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:25.375 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:25.373+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:25.446 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:25.446 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:25.447 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:25.447 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:25.455 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:25.499 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:25.497+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:25.579 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:25.577+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:25.581 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:25.581 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:25.581 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:25.590 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:25.682 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:25.681+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:25.785 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:25.783+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:25.804 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 134.94184451999999, 2024-08-30T16:01:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:01:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.805 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:01:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:01:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:01:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:01:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:01:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:01:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:01:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:01:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:01:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:01:25.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:01:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:01:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:01:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:01:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:01:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:01:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:01:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:01:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:01:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:01:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:01:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:01:25.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:01:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:01:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:25.811 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 134.0254285, 2024-08-30T16:01:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:01:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:01:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:01:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:01:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:01:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:01:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:01:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.816 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 134.02372394700001, 2024-08-30T16:01:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:01:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:01:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:01:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:01:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:01:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:01:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:01:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:01:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:01:25.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:01:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:01:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:01:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:01:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:01:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:01:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:01:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:01:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:01:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:01:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:01:25.821 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 133.60427014000001, 2024-08-30T16:01:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:01:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.822 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:01:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:01:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:01:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:01:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:01:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:01:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:01:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:01:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:01:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:01:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:01:25.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:01:25.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:01:25.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:01:25.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:01:25.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:01:25.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:01:25.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:01:25.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:01:25.827 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.827 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.827 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.828 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.828 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:25.828 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 133.58738196600001, 2024-08-30T16:01:25.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:01:25.828 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.828 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.828 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.828 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.828 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.828 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:01:25.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.828 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:25.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:25.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:01:25.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:01:25.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:01:25.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:01:25.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:01:25.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:01:25.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:01:25.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:01:25.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:01:25.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:01:25.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:01:25.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:01:25.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:01:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:01:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:01:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:01:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:01:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:01:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 133.57438189699999, 2024-08-30T16:01:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:01:25.833 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:01:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:01:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:01:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:01:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:01:25.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:01:25.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:01:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:01:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:01:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:01:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:01:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:01:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:01:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:01:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:25.838 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 133.571889605, 2024-08-30T16:01:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:01:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:01:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:01:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.840 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:01:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:01:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:01:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:01:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:01:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:01:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:01:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:01:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:01:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:01:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:01:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:01:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:01:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:01:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:01:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:01:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.843 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 133.57113215699999, 2024-08-30T16:01:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:01:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.844 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:01:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:01:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:01:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:01:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:01:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:01:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:01:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:01:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:01:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:01:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:01:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:01:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:01:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:01:25.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:01:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:01:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:01:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:01:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:01:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.849 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 133.57039970700001, 2024-08-30T16:01:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:01:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:01:25.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:01:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:01:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:01:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:01:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:01:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:01:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:01:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:01:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:01:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:01:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:01:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:01:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:01:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:01:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:01:25.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:01:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:01:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:25.855 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 133.567860908, 2024-08-30T16:01:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:01:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:01:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:01:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:01:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:01:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:01:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:01:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:01:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:01:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:01:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:01:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:01:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:01:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:01:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:01:25.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:01:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:01:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:01:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:01:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 133.56711290699999, 2024-08-30T16:01:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:01:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:01:25.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:01:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:01:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:01:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:01:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:01:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:01:25.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:01:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:01:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:01:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:01:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:01:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:01:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:01:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:01:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:01:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:01:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:01:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:01:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.794996435, 2024-08-30T16:01:25.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:01:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:01:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:01:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:01:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:01:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:01:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:01:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:01:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:01:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:01:25.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:01:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:01:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:01:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:01:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:01:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:01:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:01:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:01:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:01:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:01:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.871 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.792939289, 2024-08-30T16:01:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:01:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.872 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:01:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:01:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:01:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:01:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:01:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:01:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:01:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:01:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:01:25.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:01:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:01:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:01:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:01:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:01:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:01:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:01:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:01:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:01:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.529994789, 2024-08-30T16:01:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:01:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:01:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.878 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:01:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:01:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:01:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:01:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:01:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:01:25.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:01:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:01:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:01:25.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:01:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:01:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:01:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:01:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:01:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:01:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:01:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:01:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:01:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.528903286, 2024-08-30T16:01:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:01:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:01:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:01:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:25.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:01:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:01:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:01:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:01:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:01:25.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:01:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:01:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:01:25.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:01:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:01:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:01:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:01:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:01:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:01:25.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:01:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:01:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.52815370100001, 2024-08-30T16:01:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:01:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:01:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:25.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:01:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:01:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:01:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:01:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:01:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:01:25.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:01:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:01:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:01:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:01:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:01:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:01:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:01:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:01:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:01:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:01:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:01:25.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.49442216899999, 2024-08-30T16:01:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:01:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:01:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:25.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:01:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:01:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:01:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:01:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:01:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:01:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:01:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:01:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:01:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:01:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:01:25.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:01:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:01:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:01:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:01:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:01:25.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:01:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:01:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:25.902 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.49389616100001, 2024-08-30T16:01:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:01:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:01:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:25.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:01:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:01:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:01:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:01:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:01:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:01:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:01:25.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:01:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:01:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:01:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:01:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:01:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:01:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:01:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:01:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:01:25.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:01:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 132.49344059800001, 2024-08-30T16:01:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:01:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.908 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:01:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:01:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:01:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:01:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:01:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:01:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:01:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:01:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:01:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:01:25.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:01:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:01:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:01:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:01:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:01:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:01:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:01:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:01:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:01:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:01:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 127.822925282, 2024-08-30T16:01:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:01:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:01:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:25.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:25.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:01:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:01:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:01:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:01:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:01:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:01:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:01:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:01:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:01:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:01:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:01:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:01:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:01:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:01:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:01:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:25.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:01:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:01:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:25.919 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:25.919 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:25.921 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:25.921 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:25.919+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:26.022 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:26.020+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:26.085 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:26.086 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:26.086 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:26.086 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:26.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:26.123 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:26.121+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:26.223 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:26.222+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:26.258 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:26.258 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:26.258 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:26.268 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:26.324 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:26.323+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:26.426 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:26.424+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:26.439 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:26.439 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:26.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:26.439 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:26.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.439 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:01:26.439 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:26.439 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 119.909619835, 2024-08-30T16:01:26.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:01:26.439 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:26.439 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:26.440 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:26.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:26.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:26.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:26.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:26.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:26.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:26.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:26.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:01:26.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:01:26.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:26.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:01:26.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:01:26.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:26.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:01:26.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:01:26.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:26.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:01:26.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:01:26.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:26.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:01:26.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:01:26.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:26.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:01:26.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:01:26.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:26.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:01:26.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:01:26.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:26.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:01:26.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:01:26.444 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:26.445 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:26.445 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:26.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.445 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:01:26.445 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:26.445 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.791171689, 2024-08-30T16:01:26.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:01:26.445 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:26.445 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:26.445 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:26.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:26.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:26.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:26.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:26.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:26.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:26.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:26.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:01:26.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:01:26.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:26.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:01:26.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:01:26.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:26.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:01:26.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:01:26.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:26.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:01:26.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:01:26.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:26.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:01:26.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:01:26.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:26.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:01:26.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:01:26.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:26.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:01:26.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:01:26.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:26.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:01:26.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:01:26.449 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:26.449 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:26.449 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:26.449 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:26.449 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:26.450 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:26.451 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:26.527 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:26.525+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:26.616 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:26.616 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:26.616 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:26.616 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:26.625 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:26.627 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:26.626+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:26.728 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:26.726+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:26.748 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:26.749 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:26.749 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:26.749 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:26.749 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:26.757 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:26.842 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:26.840+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:26.924 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:26.925 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:26.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:26.925 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:26.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.925 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:26.925 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:26.925 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 118.546490611, 2024-08-30T16:01:26.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:01:26.925 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:26.925 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:26.925 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:26.926 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:26.926 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:26.926 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:01:26.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.926 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:26.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:26.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:26.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:26.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:26.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:26.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:26.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:26.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:01:26.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:01:26.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:26.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:01:26.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:01:26.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:26.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:01:26.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:01:26.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:26.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:01:26.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:01:26.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:26.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:01:26.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:01:26.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:26.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:26.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:26.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:01:26.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:01:26.929 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:26.930 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:26.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:26.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:26.930 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:26.930 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:26.931 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:26.930+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:26.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:27.032 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:27.031+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:27.077 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:27.077 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:27.077 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:27.077 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:27.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:27.133 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:27.132+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:27.215 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:27.215 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:27.215 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:27.215 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:27.215 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:27.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:01:27.234 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:27.233+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:27.335 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:27.334+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:27.352 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:27.353 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:27.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:27.353 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:01:27.353 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:27.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:01:27.436 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:27.434+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:27.490 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:27.490 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:27.490 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:27.490 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:27.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:01:27.541 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:27.539+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:27.638 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:27.636+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:27.667 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:27.667 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:27.667 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:27.668 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:27.668 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:27.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:01:27.739 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:27.737+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:27.809 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 126.534591217, 2024-08-30T16:01:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:01:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:01:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:01:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:01:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:01:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:01:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:01:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:01:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:01:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:01:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:01:27.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:27.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:01:27.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:01:27.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:27.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:01:27.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:01:27.813 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:27.813 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:27.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:27.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.814 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:27.814 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:27.814 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 126.261663856, 2024-08-30T16:01:27.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:01:27.814 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:27.814 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:27.814 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:27.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:27.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:27.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:27.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:27.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:27.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:27.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:27.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:01:27.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:01:27.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:27.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:01:27.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:01:27.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:27.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:01:27.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:01:27.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:27.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:01:27.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:01:27.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:27.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:01:27.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:01:27.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:27.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:01:27.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:01:27.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:27.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:27.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:27.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:01:27.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:01:27.818 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:27.818 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:27.818 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:27.818 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:27.818 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:27.818 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:27.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:01:27.839 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:27.838+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:27.919 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-08-30T16:01:27.919 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-08-30T16:01:27.940 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:27.939+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:27.951 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:27.951 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:27.951 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:27.951 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:27.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:01:28.041 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:28.040+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:28.127 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:28.127 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:28.128 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:28.128 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:28.128 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:28.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:01:28.142 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:28.140+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:28.250 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:28.249+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:28.266 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:28.266 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:28.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:28.266 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:28.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.266 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:01:28.267 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:28.267 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 122.889051622, 2024-08-30T16:01:28.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:01:28.267 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:28.267 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:28.267 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:28.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:28.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:28.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:28.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:28.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:28.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:01:28.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:01:28.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:28.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:01:28.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:01:28.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:01:28.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:28.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:01:28.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:01:28.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:28.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:01:28.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:01:28.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:28.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:01:28.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:01:28.271 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.271 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:28.271 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.271 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.271 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:28.271 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:28.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:01:28.345 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:28.343+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:28.444 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:28.445 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:28.445 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:28.445 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:28.445 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:28.444+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:28.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:01:28.547 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:28.545+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:28.620 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:28.620 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:28.620 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:28.620 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:28.620 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:28.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:01:28.672 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:28.671+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:28.748 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:28.747+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:28.785 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:28.785 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:28.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:28.785 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:28.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.785 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:28.786 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:28.786 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 124.332727291, 2024-08-30T16:01:28.786 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:01:28.786 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:28.786 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:28.786 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:28.786 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:28.786 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:28.786 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:28.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.786 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:28.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:28.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:28.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:28.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:28.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:28.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:01:28.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:01:28.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.788 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:28.788 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:01:28.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:01:28.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.788 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:28.788 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:01:28.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:01:28.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.789 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:28.789 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:01:28.789 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:01:28.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.789 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:28.789 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:01:28.789 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:01:28.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.790 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:01:28.790 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:01:28.790 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:01:28.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.790 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:28.790 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:01:28.790 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:01:28.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.790 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:28.790 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:01:28.790 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:01:28.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.791 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:01:28.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:01:28.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:01:28.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.791 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:28.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:01:28.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:01:28.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.792 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:28.792 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:01:28.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:01:28.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.792 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:28.792 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:01:28.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:01:28.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:28.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:01:28.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:01:28.793 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.793 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:28.793 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.793 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:01:28.793 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:28.793 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 123.216860956, 2024-08-30T16:01:28.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:01:28.794 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:28.794 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:28.794 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:28.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:28.794 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:28.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:28.794 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:28.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:28.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:01:28.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:28.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:28.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:01:28.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:01:28.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.796 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:28.796 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:01:28.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:01:28.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.796 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:28.796 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:01:28.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:01:28.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:28.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:01:28.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:01:28.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:28.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:01:28.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:01:28.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:28.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:01:28.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:01:28.798 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.798 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:28.798 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.798 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:28.798 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:28.798 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 123.157269458, 2024-08-30T16:01:28.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:01:28.799 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:28.799 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:28.799 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:28.799 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:28.799 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:28.799 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:01:28.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.799 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:28.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:28.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:28.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:28.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:28.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:28.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:01:28.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:01:28.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:28.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:01:28.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:01:28.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.801 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:28.801 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:01:28.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:01:28.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.801 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:28.801 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:01:28.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:01:28.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:28.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:01:28.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:01:28.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:28.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:01:28.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:01:28.802 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.802 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:28.802 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.803 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:01:28.803 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:28.803 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 123.130570771, 2024-08-30T16:01:28.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:01:28.803 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:28.803 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:28.803 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:28.803 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:28.803 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:28.803 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:28.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.804 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:28.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:28.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:28.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:28.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:28.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:28.805 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:01:28.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:28.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.805 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:28.805 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:01:28.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:01:28.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.805 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:28.805 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:01:28.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:01:28.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:28.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:01:28.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:01:28.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:28.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:01:28.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:01:28.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:28.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:01:28.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:01:28.807 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.807 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:28.807 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.807 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:28.807 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:28.807 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 122.262954243, 2024-08-30T16:01:28.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:01:28.807 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:28.807 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:28.807 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:28.808 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:28.808 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:28.808 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:28.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.808 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:28.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:28.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:28.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:28.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:28.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:28.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:01:28.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:28.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:28.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:01:28.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:01:28.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:28.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:01:28.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:01:28.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:28.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:01:28.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:01:28.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:28.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:01:28.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:01:28.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:28.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:01:28.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:01:28.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:28.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:01:28.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:01:28.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:28.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:01:28.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:01:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:01:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:01:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:01:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:01:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:28.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:01:28.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:01:28.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.814 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:28.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.814 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:01:28.814 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:28.814 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 122.103707164, 2024-08-30T16:01:28.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:01:28.814 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:28.814 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:28.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:01:28.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:01:28.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:28.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:01:28.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:01:28.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:28.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:01:28.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:01:28.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:28.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:01:28.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:01:28.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:28.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:01:28.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:01:28.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:28.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:01:28.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:01:28.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:28.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:01:28.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:01:28.818 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.818 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:28.818 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.818 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:28.818 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:28.819 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 121.827533098, 2024-08-30T16:01:28.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:01:28.819 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:28.819 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:28.819 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:28.819 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:28.819 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:28.819 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:28.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.819 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:28.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:28.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:28.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:28.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:28.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:28.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:01:28.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:28.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:28.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:01:28.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:01:28.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:28.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:01:28.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:01:28.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:28.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:01:28.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:01:28.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:28.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:01:28.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:01:28.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:28.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:01:28.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:01:28.822 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.823 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:28.823 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.823 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:28.823 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:28.823 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 121.169241519, 2024-08-30T16:01:28.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:01:28.823 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:28.823 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:28.823 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:28.823 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:28.823 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:01:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:28.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:01:28.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:28.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:28.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:01:28.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:01:28.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:28.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:01:28.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:01:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:01:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:01:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:01:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:01:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:28.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:01:28.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:01:28.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:28.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:01:28.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:01:28.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:28.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:01:28.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:01:28.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:28.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:01:28.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:01:28.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:28.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:01:28.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:01:28.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:28.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:01:28.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:01:28.829 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.829 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:28.829 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.829 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:01:28.829 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 121.07332823500001, 2024-08-30T16:01:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:01:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:28.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:01:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:01:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:01:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:28.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:01:28.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:01:28.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:28.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:01:28.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:01:28.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:28.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:01:28.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:01:28.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:28.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:01:28.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:01:28.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:28.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:01:28.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:01:28.833 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.833 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:28.833 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 61.821967723999997, 2024-08-30T16:01:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:01:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.834 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:01:28.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:01:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:01:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:01:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:01:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:01:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:01:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:01:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:01:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:01:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:01:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:01:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:01:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.8128630370000001, 2024-08-30T16:01:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:01:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:01:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:01:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:01:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:01:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:01:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:01:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:01:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:01:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:01:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:01:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:01:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:01:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:28.842 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:28.843 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:01:28.885 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:28.883+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:28.951 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:28.949+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:28.970 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:28.970 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:28.970 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:28.970 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:28.979 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:01:29.052 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:29.050+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:29.104 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:29.104 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:29.104 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:29.113 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:29.141 INFO:tasks.ceph.mon.a:Stopped 2024-08-30T16:01:29.141 INFO:tasks.mon_recovery:forming a minimal quorum for ['b', 'c', 'a'], then adding monitors 2024-08-30T16:01:29.141 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-30T16:01:29.142 DEBUG:teuthology.orchestra.run.smithi094:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-08-30T16:01:29.153 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:29.151+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:29.156 INFO:tasks.ceph.mon.b:Started 2024-08-30T16:01:29.157 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-30T16:01:29.157 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-08-30T16:01:29.163 INFO:tasks.ceph.mon.c:Started 2024-08-30T16:01:29.163 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2024-08-30T16:01:29.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:01:29.254 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:29.252+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:29.281 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:29.281 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:29.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:29.281 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:01:29.281 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:29.290 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:29.355 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:29.353+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:29.417 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:29.417 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:29.417 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:29.417 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:29.426 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:29.455 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:29.454+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:29.551 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:29.551 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:29.552 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:29.552 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:29.556 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:29.555+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:29.560 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:29.657 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:29.656+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:29.758 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:29.757+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:29.774 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:29.774 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:29.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:29.774 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:29.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.775 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:29.775 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:29.775 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 138.911989108, 2024-08-30T16:01:29.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:01:29.775 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.775 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.775 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.775 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.775 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.775 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:01:29.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.775 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:29.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:29.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:01:29.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:01:29.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:01:29.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:01:29.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:01:29.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:01:29.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:01:29.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:01:29.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:01:29.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:01:29.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:01:29.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:01:29.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:01:29.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:01:29.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:01:29.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:01:29.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:01:29.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:01:29.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:01:29.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:01:29.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:01:29.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:01:29.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:01:29.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.781 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.781 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.781 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:29.781 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:29.781 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.99557308799999, 2024-08-30T16:01:29.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:01:29.781 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.782 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.782 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.782 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.782 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.782 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:01:29.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.782 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:29.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:29.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:01:29.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:29.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:01:29.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:01:29.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:01:29.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:01:29.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:01:29.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:01:29.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:01:29.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:01:29.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:01:29.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:01:29.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:01:29.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:01:29.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:01:29.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:01:29.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:01:29.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:01:29.787 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.787 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.787 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.787 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:29.787 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:29.787 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.99386853499999, 2024-08-30T16:01:29.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:01:29.788 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.788 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.788 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.788 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.788 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.788 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:01:29.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.788 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:01:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:01:29.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:01:29.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:01:29.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:01:29.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:01:29.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:01:29.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:01:29.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:01:29.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:01:29.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:01:29.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:01:29.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:01:29.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:01:29.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:01:29.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:01:29.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:01:29.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:01:29.793 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.793 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.793 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.794 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.794 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:29.794 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.57441472799999, 2024-08-30T16:01:29.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:01:29.794 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.794 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.794 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.794 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.794 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.794 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:01:29.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.795 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:29.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:29.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:01:29.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:01:29.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:01:29.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:01:29.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:01:29.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:01:29.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:01:29.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:01:29.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:01:29.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:01:29.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:01:29.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:01:29.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:01:29.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:01:29.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:01:29.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:01:29.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:01:29.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:01:29.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.799 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.799 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.799 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:29.800 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.55752655399999, 2024-08-30T16:01:29.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:01:29.800 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.800 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.800 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.800 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.800 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.800 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:01:29.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.800 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:29.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:29.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:01:29.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:01:29.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:01:29.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:01:29.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:01:29.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:01:29.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:01:29.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:01:29.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:01:29.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:01:29.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:01:29.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:01:29.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:01:29.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:01:29.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:01:29.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:01:29.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:01:29.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:01:29.804 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.805 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.805 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.805 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.805 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:29.805 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.54452648500001, 2024-08-30T16:01:29.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:01:29.805 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.805 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.805 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.805 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.805 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.806 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:01:29.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.806 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:29.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:29.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:01:29.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:01:29.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:01:29.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:01:29.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:01:29.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:01:29.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:01:29.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:01:29.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:01:29.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:01:29.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:01:29.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:01:29.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:01:29.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:01:29.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:01:29.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:01:29.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:01:29.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:01:29.810 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.810 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.810 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.811 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.811 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:29.811 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.54203419300001, 2024-08-30T16:01:29.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:01:29.811 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.811 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.811 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.811 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.812 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.812 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:01:29.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.812 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:29.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:29.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:01:29.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:29.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:01:29.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:01:29.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:01:29.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:01:29.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:01:29.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:01:29.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:01:29.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:01:29.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:01:29.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:01:29.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:01:29.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:01:29.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:01:29.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:01:29.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:01:29.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:01:29.816 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.816 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.816 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.817 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.817 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:29.817 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.541276745, 2024-08-30T16:01:29.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:01:29.817 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.817 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.817 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.817 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.817 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.817 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:01:29.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.818 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:29.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:29.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:01:29.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:01:29.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:01:29.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:01:29.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:01:29.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:01:29.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:01:29.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:01:29.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:01:29.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:01:29.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:01:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:01:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:01:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:01:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:01:29.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:01:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:01:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:01:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.822 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.54054429499999, 2024-08-30T16:01:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:01:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:01:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:29.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:29.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:01:29.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:29.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:01:29.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:01:29.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:01:29.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:01:29.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:01:29.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:01:29.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:01:29.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:01:29.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:01:29.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:01:29.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:01:29.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:01:29.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:01:29.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:01:29.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:01:29.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:01:29.828 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.828 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.828 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.828 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.828 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:29.828 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.53800549600001, 2024-08-30T16:01:29.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:01:29.828 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.828 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.829 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.829 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.829 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.829 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:01:29.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.829 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.829 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:29.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.829 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:29.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:01:29.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:29.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:01:29.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:01:29.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:01:29.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:01:29.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:01:29.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:01:29.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:01:29.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:01:29.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:01:29.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:01:29.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:01:29.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:01:29.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:01:29.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:01:29.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:01:29.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:01:29.833 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.834 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.834 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.834 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.834 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:29.834 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 137.53725749500001, 2024-08-30T16:01:29.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:01:29.834 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.834 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.834 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.834 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.834 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.835 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:01:29.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.835 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:29.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:29.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:01:29.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:01:29.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:01:29.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:01:29.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:01:29.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:01:29.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:01:29.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:01:29.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:01:29.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:01:29.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:01:29.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:01:29.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:01:29.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:01:29.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:01:29.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:01:29.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:01:29.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:01:29.839 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.839 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.839 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.839 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.840 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:29.840 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 136.76514102300001, 2024-08-30T16:01:29.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:01:29.840 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.840 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.840 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.840 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.840 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.840 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:01:29.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.840 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.840 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:29.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:29.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.841 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.841 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.841 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:01:29.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:01:29.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:01:29.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:01:29.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:01:29.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:01:29.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:01:29.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:01:29.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:01:29.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:01:29.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:01:29.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:01:29.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:01:29.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:01:29.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:01:29.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:01:29.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:01:29.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:01:29.845 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.845 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.845 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.845 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.845 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:29.845 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 136.76308387700001, 2024-08-30T16:01:29.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:01:29.845 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.845 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.846 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.846 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.846 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.846 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:01:29.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.846 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:29.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:29.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:01:29.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:29.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:01:29.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:01:29.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:01:29.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:01:29.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:01:29.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:01:29.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:01:29.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:01:29.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:01:29.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:01:29.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:01:29.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:01:29.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:01:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:01:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:01:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:01:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.850 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:29.851 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 136.50013937700001, 2024-08-30T16:01:29.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:01:29.851 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.851 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.851 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.851 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.851 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.851 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:01:29.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.851 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:29.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:29.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:01:29.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:01:29.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:01:29.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:01:29.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:01:29.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:01:29.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:01:29.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:01:29.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:01:29.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:01:29.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:01:29.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:01:29.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:01:29.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:01:29.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:01:29.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:01:29.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.855 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:01:29.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:01:29.855 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.856 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.856 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.856 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.856 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:29.856 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 136.49904787400001, 2024-08-30T16:01:29.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:01:29.856 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.856 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.856 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.856 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.857 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.857 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:01:29.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.857 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:29.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:29.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:01:29.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:29.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:01:29.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:01:29.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:01:29.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:01:29.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:01:29.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:01:29.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:01:29.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:01:29.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:01:29.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:01:29.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:01:29.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:01:29.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:01:29.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:01:29.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.861 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.861 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:01:29.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:01:29.861 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.861 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.861 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.861 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.862 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:29.862 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 136.49829828899999, 2024-08-30T16:01:29.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:01:29.862 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.862 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.862 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.862 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.862 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.862 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:01:29.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.862 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:29.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:29.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:01:29.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:29.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:01:29.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:01:29.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:01:29.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:01:29.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:01:29.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:01:29.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:01:29.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:01:29.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:01:29.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:01:29.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:01:29.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:01:29.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:01:29.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:01:29.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:01:29.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:01:29.867 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.867 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.867 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.867 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.867 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:29.867 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 136.464566757, 2024-08-30T16:01:29.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:01:29.867 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.867 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.867 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.868 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.868 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.868 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:01:29.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.868 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:29.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:29.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:01:29.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:01:29.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:01:29.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:01:29.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:01:29.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:01:29.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:01:29.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:01:29.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:01:29.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:01:29.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:01:29.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:01:29.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:01:29.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:01:29.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:01:29.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:01:29.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:01:29.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:01:29.872 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.872 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.872 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.872 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.873 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:29.873 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 136.46404074899999, 2024-08-30T16:01:29.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:01:29.873 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.873 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.873 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.873 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.873 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.873 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:01:29.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.874 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:29.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:29.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:01:29.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:29.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:01:29.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:01:29.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:01:29.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:01:29.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:01:29.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:01:29.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:01:29.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:01:29.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:01:29.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:01:29.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:01:29.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:01:29.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:01:29.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:01:29.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:01:29.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:01:29.878 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.878 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.878 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.878 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:29.878 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:29.878 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 136.46358518599999, 2024-08-30T16:01:29.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:01:29.879 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.879 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.879 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.879 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.879 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.879 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:01:29.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.879 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:29.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:29.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:01:29.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:01:29.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:01:29.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:01:29.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:01:29.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:01:29.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:01:29.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:01:29.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:01:29.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:01:29.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:01:29.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:01:29.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:01:29.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:01:29.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:01:29.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:01:29.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:01:29.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:01:29.883 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.883 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.884 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.884 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:01:29.884 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:29.884 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 131.79306987000001, 2024-08-30T16:01:29.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:01:29.884 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:29.884 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:29.884 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:29.884 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:29.884 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:29.884 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:01:29.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.885 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:29.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:29.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:29.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:29.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:29.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:29.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:29.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:01:29.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:29.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:29.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:01:29.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:01:29.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:29.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:01:29.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:01:29.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:29.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:01:29.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:01:29.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:29.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:01:29.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:01:29.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:29.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:01:29.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:01:29.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:29.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:01:29.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:01:29.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:29.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:01:29.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:01:29.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:29.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:29.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:29.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:01:29.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:01:29.889 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.889 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.889 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.889 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:29.889 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:29.889 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:29.890 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:29.891 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:29.889+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:29.992 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:29.990+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:30.016 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:30.016 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:30.016 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:30.025 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:30.093 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:30.091+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:30.153 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:30.153 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:30.153 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:30.153 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:30.153 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:30.162 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:30.193 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:30.192+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:30.293 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:30.293 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:30.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:30.294 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:30.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.294 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:01:30.294 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:30.294 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 123.763480343, 2024-08-30T16:01:30.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:01:30.294 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:30.294 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:30.294 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:30.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:30.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:30.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:30.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:30.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:30.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:30.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:30.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:01:30.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:01:30.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:30.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:01:30.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:01:30.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:30.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:01:30.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:01:30.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:30.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:01:30.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:01:30.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:30.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:01:30.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:01:30.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:30.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:01:30.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:01:30.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:30.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:01:30.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:01:30.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:30.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:01:30.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:01:30.298 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:30.298 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:30.299 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:30.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.299 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:01:30.299 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:30.299 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 122.64503219700001, 2024-08-30T16:01:30.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:01:30.299 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:30.299 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:30.299 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:30.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:30.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:30.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:30.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:30.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:30.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:30.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:30.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:01:30.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:01:30.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:30.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:01:30.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:01:30.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:30.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:01:30.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:01:30.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:30.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:01:30.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:01:30.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:30.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:01:30.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:01:30.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:30.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:01:30.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:01:30.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:30.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:01:30.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:01:30.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:30.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:01:30.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:01:30.304 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:30.304 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:30.304 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:30.304 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:30.304 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:30.304 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:30.305 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:30.293+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:30.305 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:30.395 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:30.394+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:30.446 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:30.446 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:30.446 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:30.446 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:30.455 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:30.496 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:30.495+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:30.582 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:30.582 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:30.582 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:30.591 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:30.597 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:30.595+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:30.698 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:30.696+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:30.721 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:30.721 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:30.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:30.722 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:30.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.722 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:30.722 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:30.722 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 122.341514362, 2024-08-30T16:01:30.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:01:30.722 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:30.722 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:30.722 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:30.722 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:30.723 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:30.723 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:01:30.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.723 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:30.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:30.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:30.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:30.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:30.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:30.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:30.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:30.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:01:30.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:01:30.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:30.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:01:30.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:01:30.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:30.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:01:30.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:01:30.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:30.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:01:30.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:01:30.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:30.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:01:30.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:01:30.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:30.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:01:30.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:01:30.726 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:30.726 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:30.726 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:30.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.727 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:01:30.727 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:30.727 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 2.2285182460000001, 2024-08-30T16:01:30.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:01:30.727 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:30.727 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:30.727 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:30.727 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:30.727 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:30.727 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:01:30.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.728 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:30.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:30.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:30.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:30.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:30.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:30.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:30.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:30.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:01:30.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:30.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:30.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:01:30.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:01:30.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:30.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:01:30.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:01:30.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:30.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:01:30.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:01:30.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:30.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:01:30.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:01:30.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:30.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:30.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:30.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:01:30.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:01:30.731 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:30.731 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:30.731 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:30.731 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:30.731 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:30.731 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:30.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:30.799 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:30.797+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:30.872 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:30.872 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:30.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:30.899 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:30.897+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:31.003 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:31.002+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:31.051 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:31.051 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:31.052 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:31.052 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:31.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:01:31.104 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:31.102+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:31.206 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:31.204+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:31.230 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:31.231 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:31.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:31.231 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:01:31.231 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:31.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:01:31.307 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:31.306+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:31.368 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:31.369 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:31.369 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:31.369 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:31.377 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:01:31.408 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:31.406+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:31.509 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:31.507+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:31.547 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:31.547 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:31.547 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:31.547 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:31.547 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:31.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:01:31.614 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:31.608+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:31.688 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 130.41407697299999, 2024-08-30T16:01:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:01:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:01:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:01:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:01:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:01:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:01:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:01:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:01:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:01:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:01:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:01:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:01:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:01:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:01:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:01:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 130.14114961199999, 2024-08-30T16:01:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:01:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:01:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:01:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:01:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:01:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:01:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:01:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:01:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:01:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:01:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:01:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:01:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:01:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:01:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:01:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:31.697 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:31.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:01:31.710 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:31.708+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:31.811 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:31.809+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:31.865 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:31.866 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:31.866 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:31.866 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:31.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:01:31.912 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:31.910+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:32.005 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:32.109 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:32.109 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:32.109 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:32.109 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:32.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:01:32.111 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:32.109+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:32.211 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:32.209+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:32.281 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:32.281 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:32.282 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:32.282 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:32.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.282 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:01:32.282 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:32.282 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 126.904314519, 2024-08-30T16:01:32.282 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:01:32.282 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:32.282 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:32.282 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:32.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.282 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:32.282 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:32.283 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.283 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:32.283 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:32.283 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.283 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:32.283 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:01:32.283 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:01:32.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.283 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:32.284 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:01:32.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:01:32.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:32.284 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:01:32.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:01:32.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:32.284 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:01:32.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:01:32.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:32.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:01:32.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:01:32.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:32.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:01:32.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:01:32.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:32.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:01:32.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:01:32.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:32.286 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:01:32.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:01:32.286 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.286 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:32.286 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.286 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.286 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:32.286 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:32.290 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:01:32.312 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:32.310+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:32.413 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:32.412+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:32.419 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:32.419 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:32.419 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:32.419 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:32.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:01:32.514 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:32.513+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:32.599 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:32.599 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:32.599 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:32.600 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:32.600 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:32.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:01:32.615 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:32.613+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:32.715 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:32.714+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:32.802 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 128.35037508900001, 2024-08-30T16:01:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:01:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:32.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.805 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:32.805 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:01:32.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:01:32.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.805 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:32.805 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:01:32.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:01:32.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:32.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:01:32.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:01:32.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:32.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:01:32.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:01:32.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:32.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:01:32.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:01:32.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.807 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:01:32.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:01:32.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:01:32.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.807 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:32.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:01:32.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:01:32.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:32.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:01:32.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:01:32.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:01:32.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:01:32.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:01:32.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:32.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:01:32.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:01:32.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:32.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:01:32.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:01:32.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:32.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:01:32.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:01:32.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:32.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:01:32.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:01:32.810 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.810 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:32.810 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.810 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:01:32.810 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:32.810 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 127.234508754, 2024-08-30T16:01:32.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:01:32.810 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:32.810 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:32.810 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:32.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:32.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:32.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:32.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:32.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:32.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:01:32.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:32.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:32.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:01:32.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:01:32.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:32.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:01:32.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:01:32.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:32.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:01:32.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:01:32.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:32.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.815 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:32.815 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:32.815 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 127.174917256, 2024-08-30T16:01:32.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:01:32.815 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:32.815 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:32.815 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:32.815 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:32.815 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:32.815 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:01:32.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.815 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:32.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:32.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:32.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:32.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:32.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:32.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:01:32.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:01:32.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:01:32.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:32.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:01:32.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:01:32.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:32.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:01:32.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:01:32.818 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.818 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:32.818 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.819 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:01:32.819 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:32.819 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 127.14821856899999, 2024-08-30T16:01:32.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:01:32.819 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:32.819 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:32.819 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:32.819 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:32.819 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:32.819 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:32.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.819 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:32.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:32.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:32.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:32.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:32.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:32.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:01:32.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:32.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:32.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:01:32.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:01:32.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:32.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:01:32.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:01:32.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:32.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:01:32.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:01:32.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:32.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:01:32.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:01:32.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:32.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:01:32.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:01:32.823 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.823 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:32.823 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.823 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:32.823 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:32.823 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 126.28060204099999, 2024-08-30T16:01:32.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:01:32.824 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:32.824 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:32.824 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:32.824 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:32.824 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:32.824 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:32.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.824 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:32.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:32.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:32.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:01:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:32.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:01:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:01:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:01:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:01:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:01:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:01:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:32.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:01:32.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:01:32.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:32.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:01:32.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:01:32.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:32.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:01:32.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:01:32.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:32.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:01:32.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:01:32.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:32.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:01:32.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:01:32.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:32.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:01:32.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:01:32.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:32.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:01:32.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:01:32.829 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.829 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:32.829 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.829 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:01:32.829 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:32.829 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 126.121354962, 2024-08-30T16:01:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:01:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:32.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:01:32.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:01:32.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:32.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:01:32.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:01:32.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:32.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:01:32.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:01:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:01:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:01:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:01:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:01:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:32.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:01:32.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:01:32.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:32.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:01:32.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:01:32.833 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.833 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:32.833 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.833 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:32.833 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 125.845180896, 2024-08-30T16:01:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:01:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:32.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:01:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:32.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:01:32.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:01:32.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:32.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:01:32.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:01:32.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:01:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:01:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:01:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:01:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:01:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:01:32.837 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 125.186889317, 2024-08-30T16:01:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:01:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:32.838 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:01:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:32.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:01:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:01:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:01:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:01:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:01:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:01:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:01:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:01:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:01:32.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:01:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:01:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:01:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:01:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:32.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:01:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:01:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:01:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:01:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:01:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:01:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:01:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:01:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:01:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 125.090976033, 2024-08-30T16:01:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:01:32.844 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:32.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:01:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:01:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:01:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:01:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:01:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:01:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:01:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:01:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:01:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:01:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:01:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:01:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:01:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 65.839615522000003, 2024-08-30T16:01:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:01:32.848 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:01:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:01:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:01:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:01:32.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:01:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:01:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:01:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:01:32.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:01:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:01:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:01:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:01:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:01:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.8305108350000001, 2024-08-30T16:01:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:01:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:32.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:01:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:01:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:01:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:01:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:01:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:01:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:01:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:01:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:01:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:01:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:01:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:01:32.856 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.857 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:32.857 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.857 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:32.857 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:32.857 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:32.858 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:01:32.859 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:32.857+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:32.960 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:32.958+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:32.985 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:32.985 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:32.985 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:32.985 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:32.995 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:01:33.060 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:33.059+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:33.161 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:33.161 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:33.162 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:33.162 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:33.162 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:33.162 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:33.160+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:33.171 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:33.263 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:33.261+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:33.298 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:33.298 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:33.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:33.298 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:01:33.298 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:33.308 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:33.393 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:33.391+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:33.464 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:33.462+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:33.475 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:33.475 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:33.475 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:33.475 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:33.485 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:33.570 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:33.568+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:33.652 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:33.652 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:33.652 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:33.652 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:33.652 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:33.661 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:33.703 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:33.701+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:33.767 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:33.765+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:33.868 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:33.867+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:33.878 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:33.879 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:33.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:33.879 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:33.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.879 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:33.879 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:33.879 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 143.01397311400001, 2024-08-30T16:01:33.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:01:33.879 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.879 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.879 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.879 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.880 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.880 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:01:33.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.880 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:33.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:33.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:01:33.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:01:33.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:01:33.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:01:33.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:01:33.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:01:33.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:01:33.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:01:33.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:01:33.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:01:33.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:01:33.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:01:33.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:01:33.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:01:33.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:01:33.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.883 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.883 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.883 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:01:33.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:01:33.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:01:33.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:01:33.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:01:33.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:01:33.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:01:33.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:01:33.885 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.885 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.885 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.885 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:33.885 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:33.885 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 142.097557094, 2024-08-30T16:01:33.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:01:33.885 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.885 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.886 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.886 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.886 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.886 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:01:33.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.886 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:33.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:33.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:01:33.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:33.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:01:33.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:01:33.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:01:33.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:01:33.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:01:33.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:01:33.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.888 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.888 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:01:33.888 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:01:33.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:01:33.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:01:33.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:01:33.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:01:33.889 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.889 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.889 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.889 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:01:33.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:01:33.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:01:33.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:01:33.890 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.890 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.890 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.890 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:33.890 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:33.891 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 142.095852541, 2024-08-30T16:01:33.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:01:33.891 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.891 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.891 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.891 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:01:33.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.891 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:33.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:33.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:01:33.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:01:33.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:01:33.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:01:33.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:01:33.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:01:33.893 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:01:33.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:01:33.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:01:33.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:01:33.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:01:33.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:01:33.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:01:33.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:01:33.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:01:33.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:01:33.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:01:33.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:01:33.895 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.895 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.896 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.896 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.896 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:33.896 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 141.676398734, 2024-08-30T16:01:33.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:01:33.896 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.896 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.896 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.897 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:01:33.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.897 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:33.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.897 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.897 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:33.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:01:33.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:01:33.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:01:33.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:01:33.898 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.898 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.898 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.898 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:01:33.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:01:33.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:01:33.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:01:33.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.899 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.899 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:01:33.899 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:01:33.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:01:33.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:01:33.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:01:33.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:01:33.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:01:33.900 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:01:33.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:01:33.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:01:33.901 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.901 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.901 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.901 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.902 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:33.902 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 141.65951056, 2024-08-30T16:01:33.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:01:33.902 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.902 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.902 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.902 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.902 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.902 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:01:33.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.902 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:33.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:33.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:01:33.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:01:33.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:01:33.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:01:33.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:01:33.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:01:33.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:01:33.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:01:33.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:01:33.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:01:33.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.905 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.905 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:01:33.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:01:33.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:01:33.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:01:33.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:01:33.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:01:33.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:01:33.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:01:33.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.907 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.907 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.907 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:33.907 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 141.64651049099999, 2024-08-30T16:01:33.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:01:33.907 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.907 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.907 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.907 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.908 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.908 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:01:33.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.908 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:33.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:33.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:01:33.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:01:33.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:01:33.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:01:33.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:01:33.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:01:33.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:01:33.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:01:33.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:01:33.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:01:33.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:01:33.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:01:33.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:01:33.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:01:33.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:01:33.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:01:33.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.912 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.912 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:01:33.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:01:33.912 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.912 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.912 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.912 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.912 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:33.912 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 141.64401819899999, 2024-08-30T16:01:33.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:01:33.913 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.913 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.913 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.913 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.913 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.913 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:01:33.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.913 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:33.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:01:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:01:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:01:33.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:01:33.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:01:33.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:01:33.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:01:33.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:01:33.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:01:33.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:01:33.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:01:33.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:01:33.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:01:33.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:01:33.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:01:33.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:01:33.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:01:33.917 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.917 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.917 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.917 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.918 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:33.918 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 141.64326075100001, 2024-08-30T16:01:33.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:01:33.918 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.918 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.918 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.918 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.918 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.918 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:01:33.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.918 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.918 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.918 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:33.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:33.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:01:33.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:01:33.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:01:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:01:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:01:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:01:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:01:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:01:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:01:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:01:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:01:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:01:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:01:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:01:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:01:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:01:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:01:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:01:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.922 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.923 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.923 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:33.923 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 141.642528301, 2024-08-30T16:01:33.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:01:33.923 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.923 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.923 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.923 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.923 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.923 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:01:33.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.924 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:33.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.924 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:33.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.924 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.924 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.924 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:01:33.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:33.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:01:33.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:01:33.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:01:33.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:01:33.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:01:33.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:01:33.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:01:33.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:01:33.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:01:33.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:01:33.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:01:33.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:01:33.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:01:33.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:01:33.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:01:33.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:01:33.928 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.928 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.928 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.928 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.928 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:33.928 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 141.63998950199999, 2024-08-30T16:01:33.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:01:33.928 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.929 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.929 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.929 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.929 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.929 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:01:33.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.929 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:33.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:33.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:01:33.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:33.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:01:33.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:01:33.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:01:33.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:01:33.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:01:33.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:01:33.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.931 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.931 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:01:33.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:01:33.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:01:33.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:01:33.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:01:33.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:01:33.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:01:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:01:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:01:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:01:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.933 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 141.63924150099999, 2024-08-30T16:01:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:01:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:01:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:33.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:33.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:01:33.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:01:33.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:01:33.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:01:33.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:01:33.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:01:33.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:01:33.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:01:33.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:01:33.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:01:33.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:01:33.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:01:33.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:01:33.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:01:33.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:01:33.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:01:33.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:01:33.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:01:33.938 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.938 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.939 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.939 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.939 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:33.939 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.86712502899999, 2024-08-30T16:01:33.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:01:33.939 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.939 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.939 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.939 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.939 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.939 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:01:33.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.940 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:33.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:33.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:01:33.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:01:33.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:01:33.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:01:33.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:01:33.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:01:33.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:01:33.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:01:33.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:01:33.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:01:33.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:01:33.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:01:33.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:01:33.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:01:33.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:01:33.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:01:33.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:01:33.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:01:33.944 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.944 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.944 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.944 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.944 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:33.944 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.86506788299999, 2024-08-30T16:01:33.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:01:33.945 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.945 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.945 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.945 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.945 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.945 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:01:33.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.945 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:33.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:33.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:01:33.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:33.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:01:33.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:01:33.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:01:33.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:01:33.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:01:33.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:01:33.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:01:33.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:01:33.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:01:33.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:01:33.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:01:33.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:01:33.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:01:33.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:01:33.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:01:33.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:01:33.950 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.950 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.950 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.950 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.950 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:33.950 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.60212338299999, 2024-08-30T16:01:33.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:01:33.950 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.951 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.951 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.951 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:01:33.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.951 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:33.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:33.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:01:33.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:01:33.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:01:33.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:01:33.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:01:33.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:01:33.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:01:33.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:01:33.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:01:33.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:01:33.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:01:33.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:01:33.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:01:33.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:01:33.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:01:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:01:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:01:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:01:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.60103187999999, 2024-08-30T16:01:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:01:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:01:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:01:33.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:01:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:01:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:01:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:01:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:01:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:01:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:01:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:01:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:01:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:01:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:01:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:01:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:01:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:01:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:01:33.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:01:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.600282295, 2024-08-30T16:01:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:01:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.961 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:01:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:01:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:01:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:01:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:01:33.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:01:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:01:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:01:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:01:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:01:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:01:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:01:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:01:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:01:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:01:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:01:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:01:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:01:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.56655076300001, 2024-08-30T16:01:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:01:33.966 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:01:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:01:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:01:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:01:33.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:01:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:01:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:01:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:01:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:01:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:01:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:01:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:01:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:01:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:01:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:01:33.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:01:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:01:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:01:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:01:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.566024755, 2024-08-30T16:01:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:01:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:01:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:01:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:33.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:01:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:01:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:01:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:01:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:01:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:01:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:01:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:01:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:01:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:01:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:01:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:01:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:01:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:01:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:01:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:01:33.976 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.565569192, 2024-08-30T16:01:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:01:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.977 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:01:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:01:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:01:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:01:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:01:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:01:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:01:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:01:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:01:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:01:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:01:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:01:33.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:01:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:01:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:01:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:01:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:01:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:01:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:01:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:01:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 135.89505387599999, 2024-08-30T16:01:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:01:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:33.982 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:01:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:33.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:01:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:01:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:01:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:33.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:01:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:01:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:01:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:01:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:01:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:01:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:01:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:01:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:01:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:01:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:33.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:01:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:01:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:01:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:01:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:33.987 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:33.987 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:33.989 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:33.989 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:33.987+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:34.089 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:34.088+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:34.153 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:34.153 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:34.153 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:34.153 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:34.163 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:34.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-08-30T16:01:34.190 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:34.188+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:34.287 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:34.287 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:34.287 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:34.287 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:34.287 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:34.291 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:34.289+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:34.297 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:34.393 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:34.390+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:34.466 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:01:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 127.937049297, 2024-08-30T16:01:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:01:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:01:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:01:34.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:01:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:01:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:01:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:01:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:34.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:01:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:01:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:01:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:01:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:01:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:01:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:01:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:01:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:01:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:01:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.471 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:01:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 126.818601151, 2024-08-30T16:01:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:01:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:01:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:01:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:01:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:01:34.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:01:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:01:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:01:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:01:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:01:34.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:01:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:01:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:01:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:01:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:01:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:34.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:01:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:01:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:34.476 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:34.476 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:34.477 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:34.520 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:34.517+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:34.595 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:34.593+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:34.643 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:34.644 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:34.644 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:34.644 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:34.653 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:34.695 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:34.694+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:34.796 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:34.795+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:34.818 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:34.818 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:34.827 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:34.897 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:34.896+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:34.959 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:34.960 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:34.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:34.960 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:34.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.960 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:34.960 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:34.960 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 126.579763153, 2024-08-30T16:01:34.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:01:34.960 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:34.960 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:34.960 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:34.960 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:34.961 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:34.961 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:01:34.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.961 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:34.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:34.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:34.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:34.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:34.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:34.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:34.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:34.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:01:34.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:01:34.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:34.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:01:34.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:01:34.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:34.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:01:34.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:01:34.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:34.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:01:34.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:01:34.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:34.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:01:34.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:01:34.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:34.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:01:34.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:01:34.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:34.964 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:34.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:34.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.964 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:01:34.964 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:34.964 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 6.4667670370000003, 2024-08-30T16:01:34.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:01:34.964 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:34.964 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:34.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:34.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:34.965 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:34.965 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:01:34.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.965 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:34.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:34.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:34.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:34.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:34.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:34.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:34.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.966 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:34.966 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:01:34.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:34.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:34.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:01:34.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:01:34.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:34.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:01:34.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:01:34.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:34.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:01:34.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:01:34.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:34.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:01:34.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:01:34.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:34.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:34.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:34.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:01:34.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:01:34.968 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:34.968 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:34.968 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:34.968 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:34.968 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:34.969 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:34.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:35.015 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:35.013+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:35.098 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:35.098 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:35.098 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:35.098 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:35.099 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:35.097+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:35.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:35.199 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:35.198+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:35.275 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:35.275 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:35.275 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:35.275 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:35.275 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:35.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:01:35.301 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:35.299+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:35.402 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:35.401+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:35.412 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:35.413 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:35.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:35.413 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:01:35.413 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:35.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:01:35.504 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:35.501+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:35.595 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:35.595 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:35.595 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:35.595 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:35.603 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:35.601+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:35.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:01:35.704 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:35.702+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:35.734 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:35.735 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:35.735 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:35.735 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:35.735 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:35.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:01:35.805 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:35.803+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:35.879 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:35.879 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:35.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:35.879 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:35.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.879 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:35.879 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:35.879 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 134.60454224099999, 2024-08-30T16:01:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:01:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:01:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:01:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:01:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:01:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:01:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:01:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:01:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:01:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:01:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:01:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:01:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:01:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:01:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:01:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 134.33161487999999, 2024-08-30T16:01:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:01:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:01:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:01:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:01:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:01:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:01:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:01:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:01:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:01:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:01:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:01:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:01:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:01:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:01:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:01:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:35.888 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:35.888 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:35.888 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:35.888 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:35.888 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:35.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:01:35.930 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:35.929+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:36.006 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:36.005+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:36.058 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:36.058 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:36.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:01:36.110 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:36.109+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:36.208 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:36.206+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:36.239 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:36.239 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:36.239 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:36.239 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:36.239 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:36.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:01:36.309 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:36.307+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:36.378 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:01:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 131.001362333, 2024-08-30T16:01:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:01:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:01:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:01:36.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:01:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:01:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:01:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:01:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:01:36.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:01:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:01:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:01:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:01:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:01:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:36.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:01:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:01:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:01:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:01:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.383 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:36.383 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:36.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:01:36.429 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:36.428+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:36.477 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:07.262 33046== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:36.510 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:36.509+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:36.557 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:36.557 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:36.557 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:36.557 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:36.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:01:36.612 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:36.610+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:36.697 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:36.697 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:36.697 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:36.697 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:36.697 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:36.698 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:07.484 33046== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:36.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:01:36.747 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:36.746+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:36.813 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:36.811+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:36.865 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:07.642 38878== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:36.899 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 132.446623214, 2024-08-30T16:01:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:01:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:01:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:01:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:01:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:01:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:01:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:01:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:01:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:01:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:01:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:01:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:01:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:01:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:01:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:01:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:01:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:01:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:01:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:01:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:01:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:01:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:01:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:01:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:01:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:01:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:01:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:01:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:01:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:01:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:01:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 131.33075687900001, 2024-08-30T16:01:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:01:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:01:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:01:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:01:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:01:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:01:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:01:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:01:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:01:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:01:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:01:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:01:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:01:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:01:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 131.271165381, 2024-08-30T16:01:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:01:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:01:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:01:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:01:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:01:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:01:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:01:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:01:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:01:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:01:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:01:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:01:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:01:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:01:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:01:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 131.24446669400001, 2024-08-30T16:01:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:01:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:01:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:01:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:01:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:01:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:01:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:01:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:01:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:01:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:01:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:01:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:01:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 130.376850166, 2024-08-30T16:01:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:01:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:01:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:01:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:01:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:01:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:01:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:01:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:01:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:01:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:01:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:01:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:01:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:01:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:01:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:01:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:01:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:01:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:01:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:01:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:01:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:01:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:01:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:01:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 130.21760308699999, 2024-08-30T16:01:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:01:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:01:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:01:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:01:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:01:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:01:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:01:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:01:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:01:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:01:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:01:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:01:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:01:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:01:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:01:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 129.941429021, 2024-08-30T16:01:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:01:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:01:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:01:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:01:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:01:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:01:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:01:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:01:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:01:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:01:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:01:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:01:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 129.283137442, 2024-08-30T16:01:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:01:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:01:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:01:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:01:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:01:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:01:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:01:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:01:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:01:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:01:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:01:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:01:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:01:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:01:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:01:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:01:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:01:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:01:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:01:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:01:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:01:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:01:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:01:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:01:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 129.18722415799999, 2024-08-30T16:01:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:01:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:01:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:01:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:01:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:01:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:01:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:01:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:01:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:01:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:01:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:01:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:01:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:01:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:01:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 69.935863647000005, 2024-08-30T16:01:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:01:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:01:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:01:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:01:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:01:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:01:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:01:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:01:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:01:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:01:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:01:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:01:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:01:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:01:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9267589600000008, 2024-08-30T16:01:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:01:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:01:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:01:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:01:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:01:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:01:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:01:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:01:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:01:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:01:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:01:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:01:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:01:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:36.965 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:36.967 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:01:36.967 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:36.965+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:37.068 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:37.066+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:37.094 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:37.094 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:37.095 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:07.872 38878== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:37.105 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:01:37.169 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:37.167+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:37.230 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:37.231 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:37.239 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:37.284 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:37.282+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:37.368 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:37.368 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:37.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:37.368 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:01:37.368 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:37.370 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:37.368+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:37.377 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:37.470 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:37.468+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:37.568 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:37.568 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:37.568 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:37.568 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:37.571 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:37.568+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:37.579 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:37.672 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:37.670+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:37.703 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:37.703 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:37.704 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:37.704 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:37.704 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:37.713 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:37.772 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:37.770+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:37.873 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:37.871+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:37.934 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:37.934 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:37.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:37.934 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:37.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.934 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:37.934 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:37.934 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 147.07077224400001, 2024-08-30T16:01:37.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:01:37.935 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:37.935 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:37.935 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:37.935 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:37.935 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:37.935 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:01:37.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.935 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:37.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:37.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:37.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:37.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:37.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:37.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:01:37.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:01:37.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:37.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:01:37.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:01:37.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:37.937 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:01:37.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:01:37.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:37.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:01:37.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:01:37.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:37.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:01:37.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:01:37.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:01:37.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:01:37.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:01:37.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:37.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:01:37.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:01:37.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:37.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:01:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:01:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:01:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:01:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:01:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:01:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:37.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:01:37.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:01:37.942 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.942 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:37.942 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.942 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:37.942 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:37.942 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.154356224, 2024-08-30T16:01:37.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:01:37.942 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:37.942 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:37.943 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:37.943 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:37.943 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:37.943 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:01:37.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.943 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:37.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:37.943 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:37.943 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.943 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.943 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:01:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:01:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:01:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:37.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:01:37.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:01:37.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:37.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:01:37.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:01:37.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:37.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:01:37.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:01:37.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:37.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:01:37.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:01:37.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:37.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:01:37.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:01:37.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:37.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:01:37.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:01:37.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:37.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:01:37.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:01:37.947 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.947 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:37.947 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.947 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:37.947 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:37.948 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 146.152651671, 2024-08-30T16:01:37.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:01:37.948 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:37.948 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:37.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:37.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:37.948 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:37.948 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:01:37.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.948 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:37.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:37.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:37.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:37.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:37.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:37.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:01:37.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:01:37.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:37.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:01:37.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:01:37.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:37.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:01:37.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:01:37.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:01:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:01:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:01:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:01:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:01:37.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:01:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:01:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:01:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:01:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:01:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:37.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:01:37.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:01:37.953 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.953 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:37.953 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.953 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:37.953 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:37.953 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 145.733197864, 2024-08-30T16:01:37.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:01:37.954 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:37.954 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:37.954 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:37.954 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:37.954 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:37.954 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:01:37.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.954 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:37.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:37.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:37.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:37.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:37.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.955 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:37.955 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:01:37.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:01:37.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:37.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:01:37.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:01:37.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:37.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:01:37.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:01:37.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:37.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:01:37.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:01:37.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:37.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:01:37.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:01:37.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:37.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:01:37.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:01:37.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:37.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:01:37.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:01:37.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:37.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:01:37.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:01:37.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:37.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:01:37.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:01:37.960 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.960 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:37.960 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.961 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:37.961 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:37.961 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 145.71630969, 2024-08-30T16:01:37.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:01:37.961 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:37.961 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:37.961 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:37.961 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:37.961 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:37.961 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:01:37.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.961 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:37.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:37.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:37.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:37.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:37.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:37.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:01:37.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:01:37.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:37.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:01:37.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:01:37.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:37.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:01:37.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:01:37.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:37.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:01:37.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:01:37.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:37.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:01:37.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:01:37.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:37.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:01:37.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:01:37.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:37.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:01:37.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:01:37.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:37.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:01:37.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:01:37.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:37.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:01:37.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:01:37.965 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.965 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 145.70330962099999, 2024-08-30T16:01:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:01:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:01:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.967 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:37.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:37.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:37.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:37.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:37.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:37.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:01:37.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:01:37.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:37.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:01:37.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:01:37.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:37.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:01:37.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:01:37.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:37.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:01:37.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:01:37.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:37.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:01:37.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:01:37.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:37.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:01:37.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:01:37.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:37.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:01:37.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:01:37.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:37.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:01:37.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:01:37.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:37.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:01:37.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:01:37.971 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.971 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:37.971 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.971 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:37.971 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:37.971 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 145.70081732899999, 2024-08-30T16:01:37.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:01:37.971 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:37.971 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:37.971 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:37.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:37.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:37.972 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:01:37.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.972 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:37.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:37.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:37.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:37.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:37.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:37.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:01:37.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:37.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:37.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:01:37.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:01:37.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:37.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:01:37.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:01:37.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:37.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:01:37.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:01:37.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:37.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:01:37.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:01:37.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:37.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:01:37.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:01:37.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:37.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:01:37.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:01:37.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:37.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:01:37.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:01:37.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:37.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:01:37.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:01:37.976 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.976 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:37.976 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.976 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:37.976 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:37.977 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 145.70005988099999, 2024-08-30T16:01:37.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:01:37.977 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:37.977 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:37.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:37.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:37.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:37.977 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:01:37.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.977 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:37.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:37.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:37.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:01:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:01:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:37.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:01:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:01:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:01:37.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:01:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:01:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:01:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:01:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:01:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:01:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:01:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:01:37.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:01:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:01:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:01:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:01:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:01:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:37.982 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 145.699327431, 2024-08-30T16:01:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:01:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:01:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.983 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:37.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:01:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:01:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:01:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:01:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:01:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:01:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:01:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:01:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:01:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:01:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:01:37.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:01:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:01:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:01:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:01:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:37.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:01:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:01:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 145.69678863199999, 2024-08-30T16:01:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:01:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:37.988 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:01:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:37.989 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:01:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:01:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:01:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:01:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:01:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:01:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:01:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:01:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:01:37.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:01:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:01:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:01:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:01:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:01:37.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:01:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:01:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:01:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:37.993 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 145.69604063099999, 2024-08-30T16:01:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:01:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:01:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:01:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:01:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:01:37.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:01:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:01:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:01:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:01:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:01:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:01:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:01:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:01:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:01:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:01:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:01:37.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:01:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:01:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:01:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:01:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.92392415899999, 2024-08-30T16:01:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:01:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:01:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:37.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:01:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:01:38.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:01:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:01:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:01:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:01:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:38.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:01:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:01:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:01:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:01:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:01:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:01:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:01:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:01:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:01:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:01:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:01:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:01:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.921867013, 2024-08-30T16:01:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:01:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:38.004 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:01:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:01:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:01:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:01:38.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:01:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:01:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:01:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:01:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:01:38.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:01:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:01:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:01:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:01:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:01:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:38.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:01:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:01:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:01:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:01:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:38.009 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.65892251299999, 2024-08-30T16:01:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:01:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:01:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:38.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:01:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:01:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:38.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:01:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:01:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:01:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:01:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:01:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:01:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:01:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:01:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:01:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:01:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:01:38.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:01:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:01:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:01:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:01:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:01:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:38.014 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.65783101, 2024-08-30T16:01:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:01:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:01:38.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:01:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:01:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:01:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:01:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:01:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:01:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:01:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:01:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:01:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:01:38.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:01:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:01:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:01:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:01:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:01:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:38.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:01:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:01:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.657081425, 2024-08-30T16:01:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:01:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:38.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:01:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:38.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:01:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:01:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:01:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:38.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:01:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:01:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:01:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:01:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:01:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:01:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:01:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:01:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:01:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:01:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:01:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:01:38.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:01:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:01:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.62334989300001, 2024-08-30T16:01:38.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:01:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:01:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:01:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:01:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:01:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:01:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:01:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:01:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:01:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:01:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:01:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:01:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:01:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:01:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:01:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:01:38.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:01:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:01:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:01:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:01:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.622823885, 2024-08-30T16:01:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:01:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:01:38.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:01:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:01:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:01:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:01:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:01:38.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:01:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:01:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:01:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:01:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:01:38.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:01:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:01:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:01:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:01:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:01:38.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:01:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:01:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:38.036 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.622368322, 2024-08-30T16:01:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:01:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:01:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:01:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:01:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:38.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:01:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:01:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:01:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:01:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:01:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:01:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:01:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:01:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:01:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:01:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:01:38.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:01:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:01:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:01:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:01:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:01:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:38.041 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:01:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 139.95185300599999, 2024-08-30T16:01:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:01:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:01:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:38.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:01:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:01:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:01:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:01:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:01:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:01:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:01:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:01:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:01:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:01:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:01:38.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:01:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:01:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:01:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:01:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:01:38.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:01:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.047 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:38.047 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:38.048 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:38.049 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:38.047+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:38.149 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:38.148+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:38.214 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:38.214 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:38.214 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:38.214 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:38.223 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:38.264 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:38.262+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:38.352 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:38.350+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:38.389 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:38.389 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:38.389 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:38.389 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:38.389 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:38.399 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:38.454 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:38.451+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:38.529 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:38.530 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:38.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:38.530 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:38.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.530 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:01:38.531 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:38.531 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 131.99731750199999, 2024-08-30T16:01:38.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:01:38.531 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:38.531 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:38.531 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:38.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:38.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:38.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:38.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:38.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:38.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:01:38.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:01:38.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:38.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:01:38.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:01:38.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:38.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:01:38.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:01:38.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:38.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:01:38.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:01:38.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:38.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:01:38.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:01:38.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:38.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:01:38.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:01:38.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:38.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:01:38.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:01:38.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:38.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:01:38.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:01:38.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.536 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:38.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.537 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:01:38.537 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:38.537 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 130.878869356, 2024-08-30T16:01:38.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:01:38.537 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:38.537 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:38.537 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:38.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:38.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:38.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:38.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:38.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:38.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:38.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:01:38.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:01:38.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:38.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:01:38.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:01:38.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:38.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:01:38.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:01:38.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:38.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:01:38.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:01:38.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:38.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:01:38.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:01:38.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:38.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:01:38.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:01:38.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:38.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:01:38.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:01:38.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:38.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:38.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:38.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:01:38.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:01:38.541 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.541 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:38.542 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.542 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:38.542 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:38.542 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:38.543 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:38.554 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:38.552+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:38.656 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:38.654+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:38.683 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:38.684 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:38.684 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:38.684 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:38.695 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:38.780 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:38.778+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:38.857 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:38.855+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:38.864 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:38.864 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:38.864 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:38.864 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:38.864 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:38.873 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:38.958 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:38.956+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:39.054 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:39.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 130.67464979299999, 2024-08-30T16:01:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:01:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:01:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.055 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:01:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:01:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:01:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:01:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:01:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:01:39.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:01:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:01:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:01:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:01:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:39.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:01:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:01:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:01:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.561653677000001, 2024-08-30T16:01:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:01:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:39.059 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:01:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:39.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:01:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:39.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:01:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:01:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:01:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:01:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:39.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:01:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:01:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:01:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:01:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:01:39.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:01:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:39.064 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:39.064 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:39.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:39.066 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:39.063+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:39.167 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:39.165+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:39.194 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:39.194 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:39.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:39.268 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:39.265+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:39.330 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:39.330 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:39.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:01:39.381 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:39.379+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:39.469 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:39.468+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:39.536 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:39.536 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:39.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:39.537 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:01:39.537 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:39.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:01:39.592 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:39.589+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:39.670 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:39.668+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:39.677 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:39.677 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:39.677 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:39.677 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:39.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:01:39.772 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:39.769+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:39.873 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:39.871+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:39.874 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:39.874 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:39.874 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:39.874 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:39.874 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:39.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:01:39.973 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:39.971+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:40.058 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:40.058 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:40.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:40.058 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:40.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.058 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:40.059 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:40.059 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 138.783332645, 2024-08-30T16:01:40.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:01:40.059 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:40.059 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:40.059 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:40.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:40.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:40.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:40.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:40.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:40.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:40.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:40.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:01:40.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:01:40.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:40.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:01:40.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:01:40.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:40.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:01:40.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:01:40.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:40.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:01:40.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:01:40.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:40.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:01:40.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:01:40.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:40.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:01:40.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:01:40.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:01:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:01:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 138.510405284, 2024-08-30T16:01:40.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:01:40.064 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:40.064 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:40.064 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:40.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:40.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:40.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:40.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:40.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:40.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:40.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:40.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:01:40.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:01:40.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:40.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:01:40.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:01:40.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:40.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:01:40.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:01:40.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:40.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:01:40.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:01:40.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:40.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:01:40.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:01:40.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:40.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:01:40.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:01:40.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:40.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:01:40.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:01:40.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:40.069 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:40.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:40.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:40.069 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:40.069 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:40.070 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:01:40.074 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:40.072+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:40.175 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:40.174+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:40.197 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:40.197 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:40.197 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:40.197 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:40.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:01:40.276 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:40.274+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:40.336 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:40.336 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:40.336 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:40.337 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:40.337 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:40.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:01:40.377 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:40.375+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:40.478 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:40.477+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:40.517 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:40.517 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:40.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:40.517 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:40.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.518 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:01:40.518 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:40.518 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 135.13990693400001, 2024-08-30T16:01:40.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:01:40.518 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:40.518 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:40.518 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:40.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:40.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:40.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:40.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:40.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:40.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:40.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:40.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:01:40.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:01:40.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:40.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:01:40.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:01:40.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:40.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:01:40.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:01:40.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:40.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:01:40.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:01:40.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:40.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:01:40.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:01:40.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:40.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:01:40.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:01:40.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:40.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:01:40.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:01:40.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:40.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:40.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:01:40.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:01:40.522 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:40.522 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:40.522 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:40.522 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:40.522 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:40.522 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:40.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:01:40.579 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:40.577+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:40.654 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:40.655 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:40.655 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:40.655 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:40.663 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:01:40.707 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:40.705+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:40.780 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:40.779+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:40.793 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:40.793 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:40.793 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:40.793 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:40.793 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:40.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:01:40.888 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:40.885+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:40.998 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:40.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 136.544829295, 2024-08-30T16:01:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:01:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:40.999 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:01:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:01:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:01:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:01:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:41.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:01:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:01:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:01:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:01:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:01:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:01:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:01:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:01:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:01:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:01:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:01:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:01:41.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:01:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:01:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:01:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:01:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:01:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:01:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:01:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:01:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:01:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:01:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:01:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:01:41.005 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:01:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 135.42896296000001, 2024-08-30T16:01:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:01:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:41.006 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:41.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:01:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:01:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:01:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:01:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:01:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:01:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:01:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:01:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:01:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:01:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:01:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:01:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:01:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.010 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 135.369371462, 2024-08-30T16:01:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:01:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:01:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.011 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:41.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:01:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:01:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:01:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:01:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:01:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:01:41.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:01:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:01:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:01:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:01:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:41.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:01:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:01:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:01:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 135.34267277500001, 2024-08-30T16:01:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:01:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:41.015 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:01:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:41.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:01:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:01:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:01:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:01:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:01:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:01:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:01:41.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:01:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:01:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:01:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:41.020 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 134.475056247, 2024-08-30T16:01:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:01:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:41.021 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:01:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:41.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:01:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:01:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:01:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:01:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:01:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:01:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:01:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:01:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:41.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:01:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:01:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:01:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:01:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:01:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:01:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:01:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:01:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:01:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:01:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:01:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:01:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.028 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:01:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 134.31580916799999, 2024-08-30T16:01:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:01:41.029 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:01:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:01:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:01:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:01:41.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:01:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:01:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:01:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:01:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:41.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:01:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:01:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:01:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:01:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:01:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:01:41.033 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 134.03963510200001, 2024-08-30T16:01:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:01:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:41.034 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:41.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:01:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:41.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:01:41.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:01:41.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:41.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:01:41.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:01:41.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:41.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:01:41.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:01:41.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:41.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:01:41.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:01:41.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:41.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:01:41.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:01:41.039 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.039 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:41.039 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.040 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:41.040 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:41.040 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 133.381343523, 2024-08-30T16:01:41.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:01:41.040 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:41.040 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:41.040 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:41.040 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:41.040 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:41.040 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:01:41.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.041 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:41.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:41.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:41.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:41.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:41.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:41.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:01:41.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:41.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:41.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:01:41.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:01:41.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:41.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:01:41.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:01:41.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:41.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:01:41.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:01:41.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:41.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:01:41.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:01:41.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:41.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:01:41.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:01:41.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:41.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:01:41.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:01:41.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:41.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:01:41.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:01:41.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:41.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:01:41.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:01:41.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:41.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:01:41.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:01:41.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:41.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:01:41.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:01:41.046 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.046 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:41.046 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.046 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:01:41.046 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:41.046 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 133.28543023899999, 2024-08-30T16:01:41.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:01:41.047 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:41.047 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:41.047 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:41.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:41.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:41.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:41.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:41.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:41.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:01:41.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:41.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:41.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:01:41.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:01:41.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:41.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:01:41.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:01:41.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:41.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:01:41.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:01:41.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:41.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:01:41.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:01:41.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:41.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:01:41.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:01:41.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:41.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:01:41.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:01:41.051 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.051 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:41.051 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.051 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:41.051 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:41.051 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 74.034069728000006, 2024-08-30T16:01:41.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:01:41.052 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:41.052 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:41.052 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:41.052 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:41.052 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:41.052 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:41.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.052 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:41.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:41.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:41.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:41.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:41.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:41.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:01:41.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:01:41.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:41.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:01:41.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:01:41.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:41.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:01:41.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:01:41.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:41.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:01:41.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:01:41.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:41.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:01:41.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:01:41.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:41.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:01:41.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:01:41.056 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.056 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:41.056 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.057 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:01:41.057 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:41.057 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.024965041, 2024-08-30T16:01:41.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:01:41.057 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:41.057 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:41.057 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:41.057 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:41.057 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:41.057 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:41.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.057 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:41.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:41.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:41.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:41.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:41.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:41.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:41.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:01:41.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:01:41.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:41.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:01:41.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:01:41.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:41.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:01:41.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:01:41.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:41.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:01:41.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:01:41.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:41.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:01:41.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:01:41.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:41.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:41.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:41.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:01:41.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:01:41.061 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.061 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:41.061 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.062 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:41.062 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:41.062 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:41.062 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:40.996+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:41.063 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:01:41.104 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:41.102+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:41.184 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:41.184+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:41.191 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:41.192 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:41.192 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:41.192 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:41.201 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:01:41.285 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:41.282+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:41.368 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:41.368 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:41.368 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:41.377 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:41.422 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:41.421+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:41.487 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:41.486+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:41.502 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:41.502 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:41.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:41.502 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:01:41.503 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:41.511 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:41.589 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:41.586+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:41.677 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:41.678 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:41.678 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:41.678 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:41.687 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:41.689 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:41.687+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:41.790 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:41.790+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:41.810 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:41.810 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:41.811 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:41.811 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:41.811 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:41.820 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:41.892 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:41.890+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:41.993 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:41.992+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:42.031 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:42.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 151.17022304599999, 2024-08-30T16:01:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:01:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:01:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.032 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:01:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:01:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:01:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:01:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:01:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:01:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:01:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:01:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:01:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:01:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:01:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:01:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:01:42.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:01:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:01:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:01:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:01:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:01:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:01:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:01:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:01:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:01:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:01:42.037 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 150.253807026, 2024-08-30T16:01:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:01:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.038 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:01:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:01:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:01:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:01:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:01:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:01:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:01:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:01:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:01:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:01:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.041 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:01:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:01:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:01:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:01:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:01:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:01:42.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:01:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:01:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 150.25210247300001, 2024-08-30T16:01:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:01:42.043 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.044 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.044 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.044 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.044 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.044 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:01:42.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.044 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:42.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:42.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:01:42.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:01:42.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:01:42.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:01:42.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:01:42.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:01:42.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:01:42.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:01:42.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:01:42.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:01:42.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:01:42.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:01:42.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:01:42.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:01:42.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:01:42.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:01:42.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:01:42.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:01:42.049 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.049 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.049 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.049 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.050 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:42.050 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.83264866600001, 2024-08-30T16:01:42.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:01:42.050 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.050 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.050 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.050 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.050 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.050 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:01:42.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.050 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:42.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:42.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:01:42.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:01:42.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:01:42.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:01:42.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:01:42.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:01:42.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:01:42.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:01:42.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:01:42.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:01:42.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:01:42.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:01:42.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:01:42.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:01:42.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:01:42.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:01:42.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:01:42.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:01:42.055 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.055 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.055 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.055 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.055 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:42.055 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.81576049200001, 2024-08-30T16:01:42.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:01:42.055 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.055 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.056 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.056 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.056 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.056 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:01:42.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.056 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:42.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:42.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:01:42.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:01:42.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:01:42.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:01:42.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:01:42.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:01:42.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:01:42.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:01:42.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:01:42.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:01:42.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:01:42.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:01:42.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:01:42.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:01:42.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:01:42.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:01:42.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:01:42.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:01:42.060 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.060 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.060 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.061 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.061 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:42.061 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.802760423, 2024-08-30T16:01:42.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:01:42.061 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.061 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.061 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.061 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.061 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.061 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:01:42.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.061 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:42.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:42.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:01:42.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:01:42.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:01:42.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:01:42.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:01:42.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:01:42.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:01:42.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:01:42.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:01:42.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:01:42.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:01:42.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:01:42.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:01:42.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:01:42.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:01:42.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:01:42.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:01:42.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:01:42.066 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.066 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.066 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.066 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.066 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:42.066 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.800268131, 2024-08-30T16:01:42.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:01:42.066 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.067 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.067 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.067 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.067 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.067 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:01:42.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.067 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:42.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:42.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:01:42.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:42.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:01:42.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:01:42.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:01:42.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:01:42.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:01:42.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:01:42.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:01:42.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:01:42.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:01:42.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:01:42.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:01:42.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:01:42.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:01:42.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:01:42.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:01:42.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:01:42.071 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.071 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.071 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.071 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.072 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:42.072 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.79951068299999, 2024-08-30T16:01:42.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:01:42.072 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.072 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.072 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:01:42.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.072 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:42.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:42.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:01:42.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:01:42.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:01:42.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:01:42.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:01:42.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:01:42.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:01:42.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:01:42.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:01:42.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:01:42.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:01:42.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:01:42.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:01:42.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:01:42.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:01:42.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:01:42.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:01:42.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:01:42.077 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.077 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.077 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.077 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.077 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:42.077 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.79877823300001, 2024-08-30T16:01:42.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:01:42.077 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.077 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.077 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.078 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.078 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.078 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:01:42.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.078 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:42.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:42.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:01:42.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:42.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:01:42.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:01:42.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:01:42.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:01:42.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:01:42.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:01:42.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:01:42.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:01:42.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:01:42.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:01:42.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:01:42.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:01:42.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:01:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:01:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:01:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:01:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.082 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.083 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:42.083 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.796239434, 2024-08-30T16:01:42.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:01:42.083 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.083 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.083 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.083 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.083 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.083 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:01:42.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.083 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:42.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:42.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:01:42.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:42.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:01:42.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:01:42.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:01:42.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:01:42.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:01:42.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:01:42.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:01:42.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:01:42.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:01:42.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:01:42.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:01:42.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:01:42.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:01:42.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:01:42.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:01:42.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:01:42.088 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.088 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.088 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.088 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.088 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:42.088 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.795491433, 2024-08-30T16:01:42.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:01:42.088 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.088 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.088 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.089 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:01:42.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.089 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:42.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:42.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:01:42.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:01:42.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:01:42.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:01:42.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:01:42.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:01:42.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:01:42.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:01:42.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:01:42.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:01:42.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:01:42.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:01:42.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:01:42.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:01:42.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:01:42.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:01:42.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:01:42.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:01:42.093 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.093 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.093 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.093 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.094 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:42.094 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.023374961, 2024-08-30T16:01:42.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:01:42.094 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.094 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.094 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:01:42.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.094 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:42.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:42.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:01:42.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:01:42.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:01:42.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:01:42.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:01:42.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:01:42.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:01:42.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:01:42.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:01:42.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:01:42.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:01:42.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:01:42.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:01:42.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:01:42.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:01:42.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:01:42.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:01:42.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:01:42.099 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.099 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.099 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.099 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.099 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:42.099 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 149.021317815, 2024-08-30T16:01:42.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:01:42.099 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.099 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.100 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:01:42.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.100 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:42.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:42.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:01:42.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:42.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:01:42.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:01:42.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:01:42.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:01:42.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:01:42.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:01:42.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:01:42.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:01:42.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:01:42.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:01:42.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:01:42.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:01:42.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:01:42.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:01:42.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:01:42.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:01:42.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.104 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.104 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.105 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:42.105 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 148.758373315, 2024-08-30T16:01:42.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:01:42.105 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.105 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.105 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:01:42.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.105 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:42.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:42.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:01:42.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:01:42.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:01:42.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:01:42.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:01:42.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:01:42.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:01:42.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:01:42.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:01:42.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:01:42.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:01:42.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:01:42.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:01:42.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:01:42.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:01:42.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:01:42.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:01:42.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:01:42.109 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.109 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.110 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.110 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.110 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:42.110 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 148.757281812, 2024-08-30T16:01:42.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:01:42.110 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.110 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.110 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.110 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.110 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.110 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:01:42.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.111 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:42.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:42.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:01:42.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:42.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:01:42.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:01:42.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:01:42.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:01:42.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:01:42.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:01:42.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:01:42.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:01:42.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:01:42.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:01:42.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:01:42.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:01:42.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:01:42.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:01:42.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:01:42.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:01:42.115 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.115 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.115 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.115 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.115 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:42.115 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 148.75653222700001, 2024-08-30T16:01:42.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:01:42.115 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.115 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.116 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.116 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.116 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.116 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:01:42.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.116 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:42.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:42.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:01:42.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:42.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:01:42.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:01:42.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:01:42.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:01:42.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:01:42.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:01:42.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:01:42.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:01:42.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:01:42.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:01:42.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:01:42.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:01:42.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:01:42.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:01:42.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:01:42.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:01:42.120 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.120 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.120 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.120 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.120 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:42.121 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 148.72280069499999, 2024-08-30T16:01:42.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:01:42.121 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.121 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.121 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.121 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.121 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.121 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:01:42.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.121 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:42.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:42.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:01:42.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:01:42.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:01:42.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:01:42.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:01:42.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:01:42.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:01:42.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:01:42.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:01:42.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:01:42.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:01:42.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:01:42.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:01:42.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:01:42.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:01:42.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:01:42.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:01:42.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:01:42.125 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.125 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.126 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.126 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.126 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:42.126 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 148.72227468700001, 2024-08-30T16:01:42.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:01:42.126 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.126 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.126 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.126 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.126 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.126 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:01:42.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.127 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:42.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:42.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:01:42.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:42.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:01:42.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:01:42.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:01:42.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:01:42.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:01:42.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:01:42.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:01:42.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:01:42.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:01:42.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:01:42.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:01:42.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:01:42.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:01:42.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:01:42.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:01:42.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:01:42.131 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.131 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.131 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.131 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:42.131 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:42.131 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 148.72181912400001, 2024-08-30T16:01:42.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:01:42.131 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.131 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.132 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.132 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.132 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.132 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:01:42.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.132 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:42.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:42.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:01:42.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:01:42.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:01:42.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:01:42.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:01:42.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:01:42.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:01:42.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:01:42.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:01:42.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:01:42.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:01:42.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:01:42.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:01:42.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:01:42.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:01:42.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:01:42.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:01:42.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:01:42.136 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.136 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.136 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.136 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:01:42.137 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:42.137 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.051303808, 2024-08-30T16:01:42.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:01:42.137 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.137 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.137 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:42.137 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:42.137 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:42.137 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:01:42.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.137 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:42.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:42.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:01:42.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:42.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:01:42.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:01:42.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:01:42.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:01:42.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:01:42.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:01:42.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:01:42.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:01:42.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:01:42.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:01:42.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:42.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:01:42.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:01:42.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:01:42.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:01:42.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:01:42.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:01:42.142 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.142 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.142 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.142 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.142 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.142 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:42.143 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:42.144 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:42.143+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:42.245 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:42.244+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:42.309 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:42.309 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:42.309 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:42.309 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:42.318 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:42.360 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:42.359+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:42.447 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:42.446+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:42.482 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:42.483 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:42.483 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:42.483 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:42.483 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:42.492 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:42.548 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:42.546+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:42.649 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:42.647+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:42.662 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:42.663 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:42.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:42.663 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:42.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.663 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:01:42.664 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:42.664 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 136.132816977, 2024-08-30T16:01:42.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:01:42.664 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.664 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.665 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:42.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:42.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:01:42.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:01:42.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:01:42.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:01:42.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:01:42.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:01:42.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:01:42.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:01:42.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:01:42.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:01:42.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:01:42.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:01:42.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:01:42.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:01:42.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.670 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:01:42.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:01:42.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.670 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.671 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:01:42.671 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:42.671 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 135.01436883100001, 2024-08-30T16:01:42.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:01:42.671 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:42.671 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:42.671 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:42.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:42.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:42.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:42.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:42.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:42.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:42.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:01:42.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:01:42.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:42.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:01:42.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:01:42.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:42.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:01:42.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:01:42.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:42.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:01:42.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:01:42.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:42.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:01:42.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:01:42.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:42.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:01:42.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:01:42.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:42.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:01:42.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:01:42.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:42.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:42.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:42.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:01:42.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:01:42.679 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.679 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.679 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.679 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:42.679 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:42.680 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:42.681 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:42.750 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:42.749+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:42.809 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:42.810 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:42.810 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:42.810 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:42.819 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:42.850 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:42.848+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:42.952 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:42.951+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:42.984 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:42.984 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:42.984 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:42.984 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:42.984 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:42.994 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:43.053 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:43.051+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:43.154 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:43.151+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:43.163 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:43.163 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:43.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:43.163 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:43.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.163 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:43.163 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:43.164 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 134.78376321900001, 2024-08-30T16:01:43.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:01:43.164 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:43.164 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:43.164 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:43.164 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:43.164 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:43.164 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:01:43.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.164 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:43.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:43.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:43.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:43.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:43.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:43.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:43.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:43.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:01:43.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:01:43.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:43.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:01:43.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:01:43.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:43.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:01:43.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:01:43.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:43.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:01:43.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:01:43.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:43.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:01:43.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:01:43.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:43.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:01:43.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:01:43.167 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:43.167 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:43.167 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:43.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.168 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:01:43.168 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:43.168 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 14.670767102999999, 2024-08-30T16:01:43.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:01:43.168 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:43.168 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:43.168 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:43.168 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:43.168 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:43.168 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:01:43.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.168 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:43.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:43.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:43.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:43.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:43.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:43.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:43.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:43.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:01:43.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:43.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:43.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:01:43.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:01:43.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:43.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:01:43.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:01:43.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:43.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:01:43.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:01:43.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:43.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:01:43.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:01:43.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:43.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:43.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:43.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:01:43.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:01:43.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:43.171 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:43.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:43.172 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:43.172 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:43.172 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:43.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:43.254 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:43.252+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:43.312 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:43.313 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:43.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:43.365 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:43.361+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:43.457 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:43.454+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:43.492 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:43.493 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:43.493 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:43.493 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:43.493 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:43.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:01:43.558 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:43.557+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:43.659 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:43.656+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:43.669 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:43.669 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:43.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:43.669 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:01:43.669 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:43.678 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:01:43.762 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:43.760+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:43.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:01:43.860 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:43.859+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:43.961 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:43.958+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:44.025 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:44.025 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:44.025 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:44.025 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:44.025 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:44.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:01:44.077 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:44.073+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:44.162 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:44.162+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:44.207 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:44.207 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:44.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:44.207 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:44.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.208 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:44.208 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:44.208 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 142.93295096, 2024-08-30T16:01:44.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:01:44.208 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:44.208 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:44.208 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:44.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:44.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:44.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:44.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:44.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:44.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:44.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:44.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:01:44.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:01:44.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:44.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:01:44.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:01:44.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:44.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:01:44.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:01:44.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:44.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:01:44.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:01:44.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:44.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:01:44.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:01:44.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:44.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:01:44.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:01:44.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:44.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:01:44.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:01:44.212 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:44.212 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:44.212 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:44.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.212 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:44.212 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:44.212 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 142.660023599, 2024-08-30T16:01:44.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:01:44.212 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:44.212 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:44.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:01:44.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:01:44.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:44.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:01:44.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:01:44.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:44.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:01:44.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:01:44.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:44.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:01:44.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:01:44.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:44.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:01:44.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:01:44.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:44.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:01:44.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:01:44.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:44.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:01:44.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:01:44.216 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:44.216 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:44.216 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:44.216 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:44.216 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:44.216 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:44.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:01:44.263 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:44.263+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:44.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:01:44.399 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:44.397+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:44.469 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:44.466+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:44.482 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:44.482 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:44.482 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:44.482 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:44.482 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:44.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:01:44.566 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:44.566+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:44.661 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:44.661 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:44.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:44.661 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:44.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.662 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:01:44.662 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:44.662 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 139.28408732899999, 2024-08-30T16:01:44.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:01:44.662 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:44.662 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:44.662 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:44.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:44.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:44.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:44.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:44.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:44.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:44.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:44.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:01:44.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:01:44.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:44.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:01:44.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:01:44.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:44.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:01:44.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:01:44.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:44.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:01:44.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:01:44.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:44.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:01:44.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:01:44.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:44.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:01:44.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:01:44.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:44.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:01:44.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:01:44.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:44.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:44.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:44.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:01:44.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:01:44.666 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:44.666 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:44.666 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:44.666 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:44.666 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:44.666 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:44.668 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:44.667+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:44.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:01:44.769 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:44.768+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:44.799 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:44.799 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:44.799 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:44.799 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:44.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:01:44.870 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:44.867+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:44.934 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:44.935 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:44.935 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:44.935 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:44.935 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:44.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:01:44.984 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:44.982+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:45.071 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:45.071+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:45.136 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:45.136 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:45.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:45.136 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:45.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.136 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:45.136 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:45.136 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 140.68306832799999, 2024-08-30T16:01:45.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:01:45.136 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:45.137 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:45.137 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:45.137 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:45.137 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:45.137 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:45.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.137 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:45.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:45.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:45.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:45.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:45.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:45.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:01:45.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:01:45.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:45.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:01:45.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:01:45.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:45.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:01:45.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:01:45.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:45.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:01:45.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:01:45.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:45.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:01:45.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:01:45.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:01:45.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:01:45.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:01:45.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:45.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:01:45.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:01:45.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:45.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:01:45.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:01:45.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:01:45.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:01:45.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:01:45.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:45.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:01:45.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:01:45.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:01:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:01:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:01:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:01:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:01:45.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:01:45.144 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.144 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:45.144 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.144 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:01:45.144 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:45.144 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 139.567201993, 2024-08-30T16:01:45.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:01:45.144 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:45.144 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:45.144 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:45.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:45.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:45.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:45.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:45.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:45.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:01:45.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:45.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:45.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:01:45.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:01:45.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:45.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:01:45.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:01:45.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:45.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:01:45.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:01:45.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:45.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:01:45.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:01:45.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:45.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:01:45.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:01:45.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:45.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:01:45.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:01:45.148 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.148 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:45.148 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.148 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:45.148 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:45.148 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 139.50761049499999, 2024-08-30T16:01:45.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:01:45.148 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:45.149 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:45.149 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:45.149 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:45.149 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:45.149 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:01:45.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.149 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:45.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:45.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:45.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:01:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:01:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:01:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:01:45.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:01:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:01:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:01:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:01:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:45.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:01:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:01:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:01:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:01:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:01:45.152 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:45.153 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 139.480911808, 2024-08-30T16:01:45.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:01:45.153 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:45.153 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:45.153 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:45.153 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:45.153 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:45.153 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:45.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.153 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:45.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:45.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:45.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:45.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:45.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:45.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:01:45.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:45.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:45.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:01:45.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:01:45.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:45.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:01:45.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:01:45.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:45.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:01:45.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:01:45.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:45.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:01:45.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:01:45.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:45.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:01:45.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:01:45.156 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 138.61329527999999, 2024-08-30T16:01:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:01:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:45.157 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:45.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.158 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:45.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:45.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:45.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:45.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:45.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:45.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:01:45.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:45.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:45.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:01:45.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:01:45.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:01:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:01:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:01:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:01:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:45.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:01:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:01:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:01:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:01:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:01:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:01:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:45.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:01:45.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:01:45.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:45.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:01:45.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:01:45.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:45.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:01:45.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:01:45.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:45.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:01:45.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:01:45.163 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.163 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:45.163 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.163 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:01:45.163 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:45.163 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 138.45404820100001, 2024-08-30T16:01:45.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:01:45.163 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:45.164 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:45.164 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:45.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:45.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:45.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:45.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:45.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:45.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:01:45.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:01:45.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:45.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:01:45.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:01:45.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:45.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:01:45.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:01:45.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:45.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:01:45.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:01:45.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:45.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:01:45.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:01:45.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:45.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:01:45.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:01:45.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:45.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:01:45.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:01:45.168 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.168 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:45.168 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.168 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:45.168 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:45.168 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 138.177874135, 2024-08-30T16:01:45.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:01:45.169 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:45.169 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:45.169 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:45.169 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:45.169 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:45.169 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:45.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.169 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:45.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:45.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:45.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:45.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:45.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:45.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:01:45.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:45.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:45.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:01:45.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:01:45.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:45.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:01:45.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:01:45.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:45.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:01:45.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:01:45.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:45.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:01:45.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:01:45.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:45.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:01:45.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:01:45.172 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.172 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:45.172 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.173 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:45.173 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:45.173 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 137.51958255599999, 2024-08-30T16:01:45.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:01:45.173 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:45.173 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:45.173 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:45.173 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:45.173 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:45.173 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:01:45.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.173 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:45.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:45.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:45.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:45.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:45.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:45.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:01:45.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:45.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:45.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:01:45.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:01:45.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:45.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:01:45.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:01:45.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:45.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:01:45.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:01:45.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:45.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:01:45.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:01:45.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:45.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:01:45.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:01:45.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:45.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:01:45.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:01:45.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:45.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:01:45.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:01:45.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:45.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:01:45.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:01:45.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:45.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:01:45.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:01:45.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:45.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:01:45.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:01:45.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.178 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:45.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.179 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:01:45.179 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:45.179 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 137.42366927200001, 2024-08-30T16:01:45.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:01:45.179 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:45.179 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:45.179 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:45.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:45.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:45.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:45.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:45.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:45.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:01:45.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:45.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:45.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:01:45.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:01:45.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:45.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:01:45.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:01:45.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:45.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:01:45.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:01:45.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:45.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:01:45.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:01:45.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:01:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:01:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:01:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:01:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 78.172308760999996, 2024-08-30T16:01:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:01:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.184 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:01:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:01:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:01:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:01:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:01:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:01:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:01:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:01:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:45.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:01:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:01:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:01:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:01:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:01:45.189 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 18.163204073999999, 2024-08-30T16:01:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:01:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:45.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:01:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:01:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:45.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:01:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:01:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:01:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:01:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:01:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:01:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:45.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:01:45.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:01:45.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:45.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:45.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:45.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:01:45.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:01:45.193 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.193 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:45.193 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.193 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:45.193 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:45.193 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:45.195 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:01:45.195 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:45.193+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:45.296 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:45.294+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:45.336 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:45.336 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:45.336 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:45.336 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:45.346 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:01:45.397 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:45.397+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:45.480 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:45.481 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:45.481 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:45.481 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:45.481 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:45.490 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:45.531 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:45.530+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:45.599 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:45.597+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:45.668 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:45.668 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:45.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:45.669 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:01:45.669 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:45.678 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:45.723 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:45.722+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:45.801 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:45.801+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:45.807 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:45.807 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:45.807 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:45.807 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:45.817 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:45.902 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:45.902+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:45.984 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:45.984 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:45.984 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:45.984 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:45.984 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:45.993 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:46.003 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:46.001+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:46.104 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:46.104+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:46.205 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:46.203+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:46.244 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:46.244 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:46.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:46.244 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:46.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.244 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:46.244 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:46.244 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 155.37395979199999, 2024-08-30T16:01:46.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:01:46.244 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.245 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.245 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:01:46.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.245 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:46.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:46.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:01:46.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:01:46.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:01:46.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:01:46.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:01:46.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:01:46.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:01:46.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:01:46.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:01:46.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:01:46.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:01:46.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:01:46.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:01:46.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:01:46.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:01:46.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:01:46.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:01:46.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:01:46.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:01:46.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:01:46.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:01:46.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:01:46.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:01:46.251 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.251 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.251 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.251 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:46.251 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:46.252 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 154.45754377200001, 2024-08-30T16:01:46.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:01:46.252 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.252 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.252 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.252 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.252 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.252 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:01:46.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.252 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:46.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:46.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:01:46.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:46.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:01:46.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:01:46.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:01:46.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:01:46.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:01:46.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:01:46.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:01:46.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:01:46.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:01:46.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:01:46.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:01:46.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:01:46.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:01:46.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:01:46.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:01:46.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:01:46.257 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.257 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.257 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.257 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:46.257 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:46.257 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 154.45583921900001, 2024-08-30T16:01:46.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:01:46.257 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.258 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.258 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.258 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.258 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.258 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:01:46.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.258 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:46.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:46.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:01:46.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:01:46.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:01:46.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:01:46.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:01:46.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:01:46.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:01:46.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:01:46.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:01:46.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:01:46.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:01:46.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:01:46.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:01:46.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:01:46.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:01:46.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:01:46.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:01:46.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:01:46.263 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.263 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.263 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.263 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.263 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:46.263 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 154.03638541199999, 2024-08-30T16:01:46.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:01:46.263 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.263 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.263 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.264 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.264 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.264 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:01:46.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.264 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:46.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:46.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:01:46.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:01:46.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:01:46.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:01:46.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:01:46.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:01:46.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:01:46.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:01:46.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:01:46.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:01:46.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:01:46.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:01:46.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:01:46.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:01:46.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:01:46.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:01:46.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:01:46.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:01:46.268 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.268 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.269 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.269 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:46.269 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 154.01949723800001, 2024-08-30T16:01:46.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:01:46.269 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.269 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.269 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.269 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.269 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.269 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:01:46.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.270 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:46.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:46.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:01:46.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:01:46.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:01:46.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:01:46.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:01:46.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:01:46.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:01:46.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:01:46.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:01:46.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:01:46.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:01:46.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:01:46.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:01:46.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:01:46.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:01:46.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:01:46.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:01:46.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:01:46.275 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.275 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.275 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.275 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.275 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:46.275 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 154.006497169, 2024-08-30T16:01:46.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:01:46.275 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.275 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.275 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:01:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:46.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:01:46.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:01:46.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:01:46.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:01:46.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:01:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:01:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:01:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:01:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:01:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:01:46.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:01:46.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:01:46.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:01:46.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:01:46.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:01:46.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:01:46.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:01:46.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:01:46.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.280 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.281 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.281 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:46.281 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 154.004004877, 2024-08-30T16:01:46.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:01:46.281 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.281 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.281 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:01:46.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.281 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:46.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:46.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:01:46.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:46.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:01:46.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:01:46.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:01:46.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:01:46.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:01:46.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:01:46.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:01:46.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:01:46.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:01:46.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:01:46.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:01:46.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:01:46.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:01:46.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:01:46.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:01:46.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:01:46.287 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.287 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.287 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.287 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.287 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:46.287 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 154.003247429, 2024-08-30T16:01:46.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:01:46.288 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.288 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.288 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.288 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.288 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.288 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:01:46.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.288 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:46.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:46.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:01:46.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:01:46.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:01:46.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:01:46.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:01:46.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:01:46.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:01:46.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:01:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:01:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:01:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:01:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:01:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:01:46.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:01:46.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:01:46.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:01:46.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:01:46.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:01:46.292 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.293 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.293 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.293 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.293 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:46.293 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 154.00251497900001, 2024-08-30T16:01:46.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:01:46.293 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.293 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.293 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.294 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:01:46.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.294 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:46.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:46.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:01:46.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:46.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:01:46.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:01:46.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:01:46.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:01:46.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:01:46.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:01:46.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:01:46.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:01:46.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:01:46.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:01:46.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:01:46.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:01:46.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:01:46.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:01:46.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:01:46.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:01:46.298 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.298 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.298 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.298 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.298 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:46.299 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 153.99997618, 2024-08-30T16:01:46.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:01:46.299 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.299 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.299 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:01:46.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.299 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:46.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:46.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:01:46.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:46.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:01:46.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:01:46.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:01:46.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:01:46.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:01:46.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:01:46.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:01:46.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:01:46.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:01:46.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:01:46.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:01:46.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:01:46.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:01:46.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:01:46.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:01:46.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:01:46.303 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.304 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.304 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.304 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.304 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:46.304 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 153.999228179, 2024-08-30T16:01:46.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:01:46.304 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.304 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.304 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.304 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.305 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.305 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:01:46.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.305 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:46.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:46.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:01:46.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:01:46.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:01:46.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:01:46.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:01:46.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:01:46.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:01:46.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:01:46.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:01:46.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:01:46.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:01:46.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:01:46.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:01:46.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:01:46.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:01:46.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:01:46.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:01:46.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:01:46.309 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.309 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.309 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.309 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.310 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:46.310 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 153.22711170700001, 2024-08-30T16:01:46.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:01:46.310 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.310 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.310 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.310 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.310 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.310 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:01:46.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.310 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.310 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:46.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:46.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:01:46.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:01:46.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:01:46.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:01:46.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:01:46.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:01:46.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:01:46.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:01:46.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:01:46.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:01:46.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:01:46.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:01:46.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:01:46.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:01:46.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:01:46.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:01:46.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:01:46.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:01:46.315 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.315 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.315 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.315 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.315 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:46.315 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 153.22505456100001, 2024-08-30T16:01:46.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:01:46.315 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.315 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.315 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.316 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.316 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.316 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:01:46.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.316 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:46.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.316 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.316 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:46.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:01:46.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:46.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:01:46.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:01:46.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:01:46.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:01:46.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:01:46.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:01:46.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:01:46.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:01:46.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:01:46.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:01:46.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:01:46.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:01:46.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:01:46.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:01:46.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:01:46.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:01:46.320 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.320 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.320 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.321 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.321 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:46.321 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 152.962110061, 2024-08-30T16:01:46.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:01:46.321 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.321 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.321 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.321 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.321 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.321 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:01:46.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.321 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:46.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:46.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:01:46.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:01:46.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:01:46.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:01:46.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:01:46.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:01:46.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:01:46.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:01:46.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:01:46.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:01:46.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:01:46.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:01:46.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:01:46.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:01:46.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:01:46.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:01:46.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:01:46.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:01:46.326 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.326 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.326 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.326 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.326 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:46.326 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 152.96101855800001, 2024-08-30T16:01:46.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:01:46.326 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.326 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.326 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.327 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.327 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.327 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:01:46.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.327 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:46.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:46.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:01:46.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:46.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:01:46.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:01:46.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:01:46.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:01:46.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:01:46.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:01:46.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:01:46.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:01:46.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:01:46.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:01:46.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:01:46.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:01:46.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:01:46.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:01:46.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:01:46.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:01:46.331 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.331 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.331 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.332 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.332 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:46.332 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 152.96026897300001, 2024-08-30T16:01:46.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:01:46.332 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.332 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.332 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.332 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.332 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.332 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:01:46.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.333 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:46.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:46.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:01:46.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:46.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:01:46.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:01:46.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:01:46.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:01:46.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:01:46.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:01:46.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:01:46.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:01:46.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:01:46.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:01:46.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:01:46.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:01:46.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:01:46.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:01:46.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:01:46.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:01:46.337 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.337 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.337 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.337 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.337 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:46.337 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 152.92653744099999, 2024-08-30T16:01:46.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:01:46.337 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.338 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.338 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.338 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.338 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.338 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:01:46.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.338 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:46.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:46.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:01:46.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:01:46.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:01:46.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:01:46.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:01:46.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:01:46.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:01:46.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:01:46.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:01:46.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:01:46.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:01:46.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:01:46.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.341 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.341 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:01:46.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:01:46.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:01:46.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:01:46.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:01:46.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:01:46.342 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.342 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.342 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.343 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.343 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.343 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.343 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:46.343 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 152.92601143300001, 2024-08-30T16:01:46.343 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:01:46.343 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.343 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.343 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.343 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.343 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.343 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:01:46.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.344 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:46.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:46.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:01:46.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:46.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:01:46.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:01:46.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:01:46.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:01:46.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:01:46.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:01:46.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:01:46.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:01:46.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:01:46.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:01:46.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:01:46.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:01:46.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:01:46.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:01:46.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.348 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.348 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:01:46.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:01:46.348 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.348 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.348 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.348 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:46.348 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:46.348 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 152.92555587000001, 2024-08-30T16:01:46.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:01:46.349 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.349 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.349 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.349 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.349 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.349 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:01:46.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.349 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:46.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:46.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:01:46.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:01:46.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:01:46.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:01:46.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:01:46.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:01:46.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:01:46.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:01:46.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:01:46.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:01:46.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:01:46.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:01:46.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:01:46.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:01:46.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:01:46.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:01:46.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.353 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:01:46.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:01:46.353 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.353 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.353 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.354 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:01:46.354 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:46.354 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 148.255040554, 2024-08-30T16:01:46.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:01:46.354 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.354 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.354 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:46.354 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:46.354 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:46.354 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:01:46.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.355 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:46.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:46.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:01:46.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:46.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:01:46.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:01:46.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:01:46.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:01:46.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:01:46.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:01:46.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:01:46.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:01:46.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:01:46.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:01:46.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:46.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:01:46.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:01:46.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:01:46.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:01:46.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:01:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:01:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.359 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.359 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:46.360 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:46.361 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:46.359+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:46.461 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:46.460+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:46.528 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:46.529 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:46.539 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:46.581 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:46.579+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:46.663 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:46.661+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:46.712 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:46.712 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:46.712 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:46.712 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:46.712 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:46.721 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:46.764 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:46.763+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:46.864 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:46.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:01:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 140.33439082199999, 2024-08-30T16:01:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:01:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:01:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:01:46.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:01:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:01:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:01:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:01:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:01:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:01:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:01:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:01:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:01:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:01:46.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:01:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:01:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:01:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:01:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.869 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:01:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 139.215942676, 2024-08-30T16:01:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:01:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:46.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:01:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:01:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:01:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:01:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:01:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:01:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:01:46.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:01:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:01:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:01:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:01:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:01:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:01:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:01:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:01:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:01:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:46.874 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:46.875 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:46.875 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:46.863+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:46.876 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:46.966 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:46.964+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:47.003 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:47.003 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:47.004 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:47.004 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:47.013 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:47.067 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:47.064+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:47.125 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:01:47.125 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:01:47.168 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:47.166+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:47.180 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:47.180 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:47.181 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:47.181 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:47.181 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:47.191 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:47.218 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:01:47.218 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:01:47.269 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:47.267+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:47.321 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:01:47.321 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:01:47.360 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:47.360 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:47.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:47.360 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:47.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.360 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:47.361 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:47.361 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 138.98022510800001, 2024-08-30T16:01:47.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:01:47.361 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:47.361 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:47.361 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:47.361 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:47.361 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:47.361 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:01:47.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.361 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:47.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:47.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:47.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:47.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:47.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:47.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:47.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:47.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:01:47.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:01:47.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:47.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:01:47.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:01:47.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:47.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:01:47.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:01:47.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:47.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:01:47.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:01:47.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:47.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:01:47.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:01:47.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.364 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:47.364 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:01:47.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:01:47.365 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:47.365 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:47.365 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:47.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.365 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:01:47.365 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:47.365 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.867228992000001, 2024-08-30T16:01:47.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:01:47.365 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:47.365 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:47.365 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:47.365 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:47.366 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:47.366 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:01:47.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.366 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:47.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:47.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:47.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:47.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:47.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:47.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:47.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:47.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:01:47.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:47.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:47.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:01:47.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:01:47.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:47.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:01:47.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:01:47.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:47.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:01:47.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:01:47.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:47.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:01:47.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:01:47.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:47.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:47.369 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:47.369 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:01:47.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:01:47.369 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:47.369 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:47.369 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:47.369 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:47.369 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:47.369 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:47.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:47.371 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:47.369+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:47.467 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:01:47.467 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:01:47.491 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:47.489+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:47.498 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-08-30T16:01:47.501 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:47.501 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:47.502 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.0 to 1 (10000) 2024-08-30T16:01:47.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:47.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-08-30T16:01:47.552 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:01:47.553 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:01:47.597 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:47.596+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:47.649 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:01:47.650 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:01:47.673 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:47.671+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:47.703 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:47.703 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:47.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:01:47.736 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:01:47.737 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:01:47.774 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:47.772+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:47.818 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:01:47.818 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:01:47.875 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:47.873+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:47.883 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:47.883 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:47.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:01:47.976 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:47.975+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:48.073 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:48.073 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:48.076 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:48.075+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:48.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:01:48.155 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:01:48.169 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2024-08-30T16:01:48.169 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-30T16:01:48.169 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-08-30T16:01:48.177 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:48.175+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:48.212 INFO:tasks.ceph.mon.a:Started 2024-08-30T16:01:48.212 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-08-30T16:01:48.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:01:48.253 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:48.253 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:48.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:01:48.277 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:48.276+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:48.378 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:48.377+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:48.461 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:48.461 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:48.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:01:48.513 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:48.512+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:48.580 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:48.578+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:48.599 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:48.599 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:48.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:01:48.693 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:48.692+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:48.777 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:48.777 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:48.782 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:48.781+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:48.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:01:48.883 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:48.882+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:48.935 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:01:48.981 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:48.981 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:48.985 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:48.983+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:48.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:01:49.085 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:49.084+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:49.160 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:49.160 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:49.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:01:49.187 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:49.185+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:49.217 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.1 to 1 (10000) 2024-08-30T16:01:49.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-08-30T16:01:49.287 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:49.285+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:49.310 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:49.310 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:49.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:01:49.388 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:49.387+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:49.452 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:49.452 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:49.461 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:01:49.502 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:49.501+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:49.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-08-30T16:01:49.586 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:49.587 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:49.590 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:49.589+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:49.596 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:01:49.691 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:49.690+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:49.732 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:49.732 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:49.741 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:49.793 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:49.791+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:49.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:01:49.864 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:49.865 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:49.874 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:49.918 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:49.916+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:49.995 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:49.993+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:50.010 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:50.010 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:50.021 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:50.089 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:01:50.090 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:01:50.107 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:50.105+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:50.186 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:01:50.186 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:01:50.189 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:50.189 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:50.199 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:50.200 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:50.195+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:50.274 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:01:50.274 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:01:50.297 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:50.296+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:50.323 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:50.324 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:50.333 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:50.365 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:01:50.365 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:01:50.398 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:50.397+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:50.461 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:50.461 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:01:50.467 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.2 to 1 (10000) 2024-08-30T16:01:50.472 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:50.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-08-30T16:01:50.517 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:50.515+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-08-30T16:01:50.554 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:01:50.554 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:01:50.600 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:50.598+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:50.656 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:01:50.656 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:01:50.657 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:50.657 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:50.657 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:50.657 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:50.657 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:50.668 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:50.701 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:50.700+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:50.745 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:01:50.746 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:01:50.824 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:50.822+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:50.841 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:50.841 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:50.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:50.841 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:50.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.842 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:01:50.842 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:50.842 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 144.31169407600001, 2024-08-30T16:01:50.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:01:50.842 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:50.842 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:50.842 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:50.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:50.842 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:50.842 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:50.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:50.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:50.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:50.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:50.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:01:50.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:01:50.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:50.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:01:50.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:01:50.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:50.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:01:50.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:01:50.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:50.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:01:50.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:01:50.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:50.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:01:50.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:01:50.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:50.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:01:50.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:01:50.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:50.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:01:50.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:01:50.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:50.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:01:50.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:01:50.846 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:50.846 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:50.847 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:50.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.847 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:01:50.847 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:50.847 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 143.19324592999999, 2024-08-30T16:01:50.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:01:50.847 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:50.847 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:50.847 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:50.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:50.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:50.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:50.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:50.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:50.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:50.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:50.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:01:50.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:01:50.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:50.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:01:50.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:01:50.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.849 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:50.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:01:50.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:01:50.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:50.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:01:50.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:01:50.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:50.850 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:01:50.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:01:50.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.850 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:50.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:01:50.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:01:50.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:50.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:01:50.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:01:50.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:50.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:50.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:50.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:01:50.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:01:50.851 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:50.851 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:50.852 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:50.852 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:50.852 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:50.852 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:50.853 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:50.938 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:50.936+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:51.003 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:51.001+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:51.016 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:51.017 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:51.017 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:51.017 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:51.025 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:51.103 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:51.102+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:51.191 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:51.191 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:51.191 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:51.201 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:51.204 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:51.202+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:51.305 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:51.303+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:51.328 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:51.328 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:51.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:51.328 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:51.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 142.94852105499999, 2024-08-30T16:01:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:01:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:01:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:51.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:51.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:01:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:01:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:01:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:01:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:51.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:01:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:01:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:01:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:01:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:51.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:01:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:01:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:01:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:01:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:51.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.334 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:01:51.334 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:51.334 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.835524938999999, 2024-08-30T16:01:51.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:01:51.334 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:01:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.335 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:01:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:01:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:01:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:51.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:01:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:01:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:01:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:01:51.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:01:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:01:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:01:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:01:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:51.341 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:51.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:51.406 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:51.405+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:51.465 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.3 to 1 (10000) 2024-08-30T16:01:51.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-08-30T16:01:51.499 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:51.499 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:51.499 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:51.507 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:51.506+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:51.512 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:51.608 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:51.607+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:51.686 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:51.686 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:51.686 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:51.686 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:51.686 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:51.695 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:01:51.709 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:51.707+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:51.809 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:51.808+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:51.862 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:51.862 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:51.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:51.862 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:01:51.862 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:51.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:01:51.910 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:51.909+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:51.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:01:52.030 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:52.029+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:52.044 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:52.044 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:52.044 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:52.045 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:52.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:01:52.112 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:52.110+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:52.213 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:52.212+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:52.223 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:52.223 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:52.223 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:52.224 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:52.224 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:52.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:01:52.314 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:52.312+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:52.367 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 151.092879893, 2024-08-30T16:01:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:01:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:01:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:01:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:01:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:01:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:01:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:01:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:01:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:01:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:01:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:01:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:01:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:01:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:01:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:01:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 150.819952532, 2024-08-30T16:01:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:01:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:01:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:01:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:01:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:01:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:01:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:01:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:01:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:01:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:01:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:01:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:01:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:01:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:01:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:01:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:52.378 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:52.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:01:52.422 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:52.421+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:52.481 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.4 to 1 (10000) 2024-08-30T16:01:52.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-08-30T16:01:52.524 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:52.524 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:52.524 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:52.524 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:52.527 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:52.524+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:52.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:01:52.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-08-30T16:01:52.624 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:52.622+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:52.644 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:01:52.717 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:52.716+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:52.719 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:52.720 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:52.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:01:52.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:01:52.821 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:52.819+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:52.915 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:52.915 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:52.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:52.915 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:52.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.915 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:01:52.915 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:52.916 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 147.53750294, 2024-08-30T16:01:52.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:01:52.916 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:52.916 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:52.916 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:52.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:52.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:52.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:52.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:52.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:52.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:52.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:52.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:01:52.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:01:52.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:52.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:01:52.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:01:52.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:52.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:01:52.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:01:52.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:52.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:01:52.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:01:52.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:52.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:01:52.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:01:52.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:52.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:01:52.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:01:52.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:52.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:01:52.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:01:52.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:52.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:52.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:52.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:01:52.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:01:52.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:52.920 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:52.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:52.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:52.920 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:52.920 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:52.921 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:52.920+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:52.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:01:53.023 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:53.021+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:53.062 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:01:53.063 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:01:53.124 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:53.122+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:53.158 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:53.159 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:53.159 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:53.159 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:53.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:01:53.224 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:53.222+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:53.271 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:01:53.272 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:01:53.312 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:53.312 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:53.312 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:53.312 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:53.312 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:53.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:01:53.328 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:53.326+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:53.376 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:01:53.377 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:01:53.429 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:53.427+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:53.502 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:01:53.503 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:01:53.503 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:53.503 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:53.513 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:01:53.555 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:53.553+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:53.586 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.5 to 1 (10000) 2024-08-30T16:01:53.593 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-08-30T16:01:53.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-08-30T16:01:53.608 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:01:53.608 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:01:53.631 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:53.629+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:53.661 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:53.661 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:53.671 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:01:53.718 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:01:53.718 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:01:53.731 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:53.730+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:53.810 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:01:53.810 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:01:53.844 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:53.834+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:53.850 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:53.850 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:53.859 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:53.896 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:01:53.896 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:01:53.949 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:53.947+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:54.023 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:54.023 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:54.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:54.034 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:54.032+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:54.135 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:54.133+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:54.158 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:54.159 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:54.167 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:54.236 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:54.235+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:54.289 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:54.290 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:54.298 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:54.341 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:54.338+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:54.439 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:54.437+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:54.461 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:54.461 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:54.470 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:54.539 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:54.538+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:54.636 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:54.636 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:54.640 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:54.639+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:54.645 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:54.685 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.7 to 1 (10000) 2024-08-30T16:01:54.700 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-08-30T16:01:54.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:01:54.741 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:54.739+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:54.779 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:54.779 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:54.789 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:54.841 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:01:54.840+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:01:54.943 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:54.941+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:54.952 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:54.952 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:54.963 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:55.047 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:55.045+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:55.126 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:55.127 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:55.135 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:55.178 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:55.175+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:55.245 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:55.243+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:55.298 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:55.299 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:55.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:55.345 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:55.344+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:01:55.348 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:01:55.361 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-08-30T16:01:54.681798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":397859,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":192332,"kb_used_data":3168,"kb_used_omap":10,"kb_used_meta":189045,"kb_avail":660410548,"statfs":{"total":676457349120,"available":676260401152,"internally_reserved":0,"allocated":3244032,"data_stored":2165394,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":11157,"internal_metadata":193582187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":96,"apply_latency_ms":96,"commit_latency_ns":96000000,"apply_latency_ns":96000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.152160"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":45,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-08-30T16:01:52.857148+0000","last_change":"2024-08-30T15:59:27.646756+0000","last_active":"2024-08-30T16:01:52.857148+0000","last_peered":"2024-08-30T16:01:52.857148+0000","last_clean":"2024-08-30T16:01:52.857148+0000","last_became_active":"2024-08-30T15:59:25.342321+0000","last_became_peered":"2024-08-30T15:59:25.342321+0000","last_unstale":"2024-08-30T16:01:52.857148+0000","last_undegraded":"2024-08-30T16:01:52.857148+0000","last_fullsized":"2024-08-30T16:01:52.857148+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:53:42.927368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.116914037,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":43,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-08-30T16:01:50.101982+0000","last_change":"2024-08-30T15:59:28.156737+0000","last_active":"2024-08-30T16:01:50.101982+0000","last_peered":"2024-08-30T16:01:50.101982+0000","last_clean":"2024-08-30T16:01:50.101982+0000","last_became_active":"2024-08-30T15:59:25.662524+0000","last_became_peered":"2024-08-30T15:59:25.662524+0000","last_unstale":"2024-08-30T16:01:50.101982+0000","last_undegraded":"2024-08-30T16:01:50.101982+0000","last_fullsized":"2024-08-30T16:01:50.101982+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:05:30.092449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064329163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-08-30T16:01:52.851234+0000","last_change":"2024-08-30T15:59:27.633478+0000","last_active":"2024-08-30T16:01:52.851234+0000","last_peered":"2024-08-30T16:01:52.851234+0000","last_clean":"2024-08-30T16:01:52.851234+0000","last_became_active":"2024-08-30T15:59:24.572730+0000","last_became_peered":"2024-08-30T15:59:24.572730+0000","last_unstale":"2024-08-30T16:01:52.851234+0000","last_undegraded":"2024-08-30T16:01:52.851234+0000","last_fullsized":"2024-08-30T16:01:52.851234+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:36:20.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-08-30T16:01:53.103146+0000","last_change":"2024-08-30T16:01:00.718949+0000","last_active":"2024-08-30T16:01:53.103146+0000","last_peered":"2024-08-30T16:01:53.103146+0000","last_clean":"2024-08-30T16:01:53.103146+0000","last_became_active":"2024-08-30T16:01:00.713130+0000","last_became_peered":"2024-08-30T16:01:00.713130+0000","last_unstale":"2024-08-30T16:01:53.103146+0000","last_undegraded":"2024-08-30T16:01:53.103146+0000","last_fullsized":"2024-08-30T16:01:53.103146+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T03:38:51.475159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":59,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-08-30T16:01:52.838341+0000","last_change":"2024-08-30T16:01:01.435839+0000","last_active":"2024-08-30T16:01:52.838341+0000","last_peered":"2024-08-30T16:01:52.838341+0000","last_clean":"2024-08-30T16:01:52.838341+0000","last_became_active":"2024-08-30T16:01:00.974833+0000","last_became_peered":"2024-08-30T16:01:00.974833+0000","last_unstale":"2024-08-30T16:01:52.838341+0000","last_undegraded":"2024-08-30T16:01:52.838341+0000","last_fullsized":"2024-08-30T16:01:52.838341+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:09:38.575833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-08-30T16:01:52.832419+0000","last_change":"2024-08-30T15:59:27.825532+0000","last_active":"2024-08-30T16:01:52.832419+0000","last_peered":"2024-08-30T16:01:52.832419+0000","last_clean":"2024-08-30T16:01:52.832419+0000","last_became_active":"2024-08-30T15:59:25.469390+0000","last_became_peered":"2024-08-30T15:59:25.469390+0000","last_unstale":"2024-08-30T16:01:52.832419+0000","last_undegraded":"2024-08-30T16:01:52.832419+0000","last_fullsized":"2024-08-30T16:01:52.832419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T17:46:39.688911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":45,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-08-30T16:01:52.828583+0000","last_change":"2024-08-30T15:59:27.583986+0000","last_active":"2024-08-30T16:01:52.828583+0000","last_peered":"2024-08-30T16:01:52.828583+0000","last_clean":"2024-08-30T16:01:52.828583+0000","last_became_active":"2024-08-30T15:59:24.528536+0000","last_became_peered":"2024-08-30T15:59:24.528536+0000","last_unstale":"2024-08-30T16:01:52.828583+0000","last_undegraded":"2024-08-30T16:01:52.828583+0000","last_fullsized":"2024-08-30T16:01:52.828583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:18:40.602710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-08-30T16:01:52.842944+0000","last_change":"2024-08-30T16:01:01.316424+0000","last_active":"2024-08-30T16:01:52.842944+0000","last_peered":"2024-08-30T16:01:52.842944+0000","last_clean":"2024-08-30T16:01:52.842944+0000","last_became_active":"2024-08-30T16:01:00.806973+0000","last_became_peered":"2024-08-30T16:01:00.806973+0000","last_unstale":"2024-08-30T16:01:52.842944+0000","last_undegraded":"2024-08-30T16:01:52.842944+0000","last_fullsized":"2024-08-30T16:01:52.842944+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T23:28:24.797655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":43,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-08-30T16:01:20.315857+0000","last_change":"2024-08-30T15:59:21.623783+0000","last_active":"2024-08-30T16:01:20.315857+0000","last_peered":"2024-08-30T16:01:20.315857+0000","last_clean":"2024-08-30T16:01:20.315857+0000","last_became_active":"2024-08-30T15:59:20.848924+0000","last_became_peered":"2024-08-30T15:59:20.848924+0000","last_unstale":"2024-08-30T16:01:20.315857+0000","last_undegraded":"2024-08-30T16:01:20.315857+0000","last_fullsized":"2024-08-30T16:01:20.315857+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:14:52.575421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":795680,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280551,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":466944,"data_stored":581861,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":10.308999999999999,"5min":13.598000000000001,"15min":14.146000000000001},"min":{"1min":5.585,"5min":5.585,"15min":5.585},"max":{"1min":23.088000000000001,"5min":57.177999999999997,"15min":57.177999999999997},"last":16.757000000000001},{"interface":"front","average":{"1min":11.601000000000001,"5min":12.992000000000001,"15min":13.224},"min":{"1min":5.0510000000000002,"5min":5.0510000000000002,"15min":5.0510000000000002},"max":{"1min":29.195,"5min":58.024999999999999,"15min":58.024999999999999},"last":23.323}]},{"osd":2,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":11.406000000000001,"5min":10.624000000000001,"15min":10.494},"min":{"1min":6.117,"5min":6.117,"15min":6.117},"max":{"1min":73.399000000000001,"5min":73.399000000000001,"15min":73.399000000000001},"last":21.949000000000002},{"interface":"front","average":{"1min":16.138000000000002,"5min":15.289999999999999,"15min":15.148},"min":{"1min":9.7469999999999999,"5min":6.9139999999999997,"15min":6.9139999999999997},"max":{"1min":75.120000000000005,"5min":75.120000000000005,"15min":75.120000000000005},"last":13.269}]},{"osd":3,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":14.863,"5min":17.103000000000002,"15min":17.477},"min":{"1min":7.0410000000000004,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":21.734000000000002,"5min":43.067999999999998,"15min":43.067999999999998},"last":20.475000000000001},{"interface":"front","average":{"1min":17.216000000000001,"5min":15.385,"15min":15.08},"min":{"1min":7.6619999999999999,"5min":7.6619999999999999,"15min":7.6619999999999999},"max":{"1min":104.38200000000001,"5min":104.38200000000001,"15min":104.38200000000001},"last":18.754000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":15.086,"5min":14.839,"15min":14.798},"min":{"1min":7.6950000000000003,"5min":7.6950000000000003,"15min":7.6950000000000003},"max":{"1min":30.754999999999999,"5min":30.754999999999999,"15min":30.754999999999999},"last":28.326000000000001},{"interface":"front","average":{"1min":13.462999999999999,"5min":13.144,"15min":13.090999999999999},"min":{"1min":7.0039999999999996,"5min":7.0039999999999996,"15min":7.0039999999999996},"max":{"1min":29.818999999999999,"5min":29.818999999999999,"15min":29.818999999999999},"last":15.907}]},{"osd":5,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":12.845000000000001,"5min":16.100999999999999,"15min":16.643000000000001},"min":{"1min":8.2170000000000005,"5min":8.1950000000000003,"15min":8.1950000000000003},"max":{"1min":29.474,"5min":46.100000000000001,"15min":46.100000000000001},"last":24.623000000000001},{"interface":"front","average":{"1min":17.539999999999999,"5min":19.335999999999999,"15min":19.635999999999999},"min":{"1min":9.8179999999999996,"5min":9.8179999999999996,"15min":9.8179999999999996},"max":{"1min":31.367000000000001,"5min":45.216999999999999,"15min":45.216999999999999},"last":9.5800000000000001}]},{"osd":6,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":17.210999999999999,"5min":17.657,"15min":17.731000000000002},"min":{"1min":9.1929999999999996,"5min":9.1929999999999996,"15min":9.1929999999999996},"max":{"1min":36.987000000000002,"5min":58.201999999999998,"15min":58.201999999999998},"last":29.425999999999998},{"interface":"front","average":{"1min":17.010000000000002,"5min":17.103999999999999,"15min":17.120000000000001},"min":{"1min":9.8279999999999994,"5min":8.8390000000000004,"15min":8.8390000000000004},"max":{"1min":35.582999999999998,"5min":37.253,"15min":37.253},"last":25.949000000000002}]},{"osd":7,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":17.559000000000001,"5min":19.727,"15min":20.088999999999999},"min":{"1min":12.42,"5min":11.590999999999999,"15min":11.590999999999999},"max":{"1min":29.251999999999999,"5min":40.990000000000002,"15min":40.990000000000002},"last":10.456},{"interface":"front","average":{"1min":17.387,"5min":19.103000000000002,"15min":19.388999999999999},"min":{"1min":8.3640000000000008,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":32.363,"5min":59.079000000000001,"15min":59.079000000000001},"last":31.052}]}]},{"osd":7,"up_from":21,"seq":90194313255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":201194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":8.8529999999999998,"5min":8.9410000000000007,"15min":8.9559999999999995},"min":{"1min":4.899,"5min":4.7169999999999996,"15min":4.7169999999999996},"max":{"1min":17.946000000000002,"5min":30.600999999999999,"15min":30.600999999999999},"last":25.759},{"interface":"front","average":{"1min":8.7629999999999999,"5min":9.0510000000000002,"15min":9.0990000000000002},"min":{"1min":5.1440000000000001,"5min":5.0659999999999998,"15min":5.0659999999999998},"max":{"1min":21.960999999999999,"5min":21.960999999999999,"15min":21.960999999999999},"last":28.077000000000002}]},{"osd":1,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":16.844000000000001,"5min":12.037000000000001,"15min":11.236000000000001},"min":{"1min":7.6989999999999998,"5min":6.4180000000000001,"15min":6.4180000000000001},"max":{"1min":83.581000000000003,"5min":83.581000000000003,"15min":83.581000000000003},"last":5.7990000000000004},{"interface":"front","average":{"1min":14.843999999999999,"5min":12.894,"15min":12.569000000000001},"min":{"1min":7.7380000000000004,"5min":7.2590000000000003,"15min":7.2590000000000003},"max":{"1min":81.722999999999999,"5min":81.722999999999999,"15min":81.722999999999999},"last":8.8309999999999995}]},{"osd":2,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":14.759,"5min":13.823,"15min":13.667},"min":{"1min":9.6859999999999999,"5min":6.867,"15min":6.867},"max":{"1min":29.75,"5min":29.75,"15min":29.75},"last":29.251000000000001},{"interface":"front","average":{"1min":13.172000000000001,"5min":16.649000000000001,"15min":17.228999999999999},"min":{"1min":6.9800000000000004,"5min":6.9800000000000004,"15min":6.9800000000000004},"max":{"1min":30.853999999999999,"5min":36.134999999999998,"15min":36.134999999999998},"last":12.957000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":20.297000000000001,"5min":17.335999999999999,"15min":16.841999999999999},"min":{"1min":9.1980000000000004,"5min":8.8420000000000005,"15min":8.8420000000000005},"max":{"1min":65.739999999999995,"5min":65.739999999999995,"15min":65.739999999999995},"last":10.497999999999999},{"interface":"front","average":{"1min":18.071000000000002,"5min":13.746,"15min":13.025},"min":{"1min":7.4020000000000001,"5min":7.3849999999999998,"15min":7.3849999999999998},"max":{"1min":64.980000000000004,"5min":64.980000000000004,"15min":64.980000000000004},"last":9.6120000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":15.869999999999999,"5min":13.037000000000001,"15min":12.565},"min":{"1min":7.6710000000000003,"5min":7.0339999999999998,"15min":7.0339999999999998},"max":{"1min":37.746000000000002,"5min":37.746000000000002,"15min":37.746000000000002},"last":12.167},{"interface":"front","average":{"1min":16.552,"5min":17.541,"15min":17.706},"min":{"1min":11.145,"5min":9.2070000000000007,"15min":9.2070000000000007},"max":{"1min":32.526000000000003,"5min":37.393000000000001,"15min":37.393000000000001},"last":13.484}]},{"osd":5,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":12.071,"5min":11.478,"15min":11.379},"min":{"1min":6.2309999999999999,"5min":6.2309999999999999,"15min":6.2309999999999999},"max":{"1min":29.754999999999999,"5min":29.754999999999999,"15min":29.754999999999999},"last":7.3150000000000004},{"interface":"front","average":{"1min":16.251999999999999,"5min":16.863,"15min":16.965},"min":{"1min":10.015000000000001,"5min":9.8620000000000001,"15min":9.8620000000000001},"max":{"1min":38.618000000000002,"5min":38.618000000000002,"15min":38.618000000000002},"last":23.48}]},{"osd":6,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":16.870000000000001,"5min":13.715,"15min":13.189},"min":{"1min":8.9380000000000006,"5min":6.3230000000000004,"15min":6.3230000000000004},"max":{"1min":42.829999999999998,"5min":42.829999999999998,"15min":42.829999999999998},"last":11.244},{"interface":"front","average":{"1min":19.25,"5min":18.513999999999999,"15min":18.391999999999999},"min":{"1min":10.749000000000001,"5min":10.749000000000001,"15min":10.749000000000001},"max":{"1min":44.491999999999997,"5min":44.491999999999997,"15min":44.491999999999997},"last":25.225999999999999}]}]},{"osd":6,"up_from":0,"seq":90194313254,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":21,"seq":90194313254,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":201213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":8.3100000000000005,"5min":10.964,"15min":11.406000000000001},"min":{"1min":5.0890000000000004,"5min":4.29,"15min":4.29},"max":{"1min":13.206,"5min":50.500999999999998,"15min":50.500999999999998},"last":16.832000000000001},{"interface":"front","average":{"1min":9.8729999999999993,"5min":13.948,"15min":14.627000000000001},"min":{"1min":6.2069999999999999,"5min":6.2069999999999999,"15min":6.2069999999999999},"max":{"1min":25.311,"5min":51.965000000000003,"15min":51.965000000000003},"last":17.637}]},{"osd":1,"last update":"Fri Aug 30 16:01:22 2024","interfaces":[{"interface":"back","average":{"1min":14.113,"5min":18.672999999999998,"15min":19.433},"min":{"1min":8.2460000000000004,"5min":6.0229999999999997,"15min":6.0229999999999997},"max":{"1min":32.478999999999999,"5min":137.56700000000001,"15min":137.56700000000001},"last":19.919},{"interface":"front","average":{"1min":12.951000000000001,"5min":16.577999999999999,"15min":17.181999999999999},"min":{"1min":5.0060000000000002,"5min":5.0060000000000002,"15min":5.0060000000000002},"max":{"1min":23.946999999999999,"5min":136.321,"15min":136.321},"last":19.338999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":12.608000000000001,"5min":16.045999999999999,"15min":16.619},"min":{"1min":7.4219999999999997,"5min":7.4219999999999997,"15min":7.4219999999999997},"max":{"1min":25.902000000000001,"5min":73.831000000000003,"15min":73.831000000000003},"last":43.869999999999997},{"interface":"front","average":{"1min":12.295999999999999,"5min":15.694000000000001,"15min":16.260000000000002},"min":{"1min":7.5670000000000002,"5min":7.4169999999999998,"15min":7.4169999999999998},"max":{"1min":20.948,"5min":71.831999999999994,"15min":71.831999999999994},"last":8.1069999999999993}]},{"osd":4,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":15.612,"5min":28.602,"15min":30.766999999999999},"min":{"1min":8.2460000000000004,"5min":8.2460000000000004,"15min":8.2460000000000004},"max":{"1min":29.398,"5min":342.34500000000003,"15min":342.34500000000003},"last":43.235999999999997},{"interface":"front","average":{"1min":14.583,"5min":19.834,"15min":20.709},"min":{"1min":8.0299999999999994,"5min":8.0299999999999994,"15min":8.0299999999999994},"max":{"1min":28.853000000000002,"5min":127.857,"15min":127.857},"last":63.633000000000003}]},{"osd":5,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":13.646000000000001,"5min":15.612,"15min":15.94},"min":{"1min":7.2389999999999999,"5min":7.2389999999999999,"15min":7.2389999999999999},"max":{"1min":26.850999999999999,"5min":29.577999999999999,"15min":29.577999999999999},"last":33.68},{"interface":"front","average":{"1min":16.716999999999999,"5min":18.940999999999999,"15min":19.312000000000001},"min":{"1min":10.073,"5min":7.2370000000000001,"15min":7.2370000000000001},"max":{"1min":34.149000000000001,"5min":39.939,"15min":39.939},"last":58.212000000000003}]},{"osd":6,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":16.007999999999999,"5min":29.869,"15min":32.179000000000002},"min":{"1min":11.974,"5min":8.6630000000000003,"15min":8.6630000000000003},"max":{"1min":33.237000000000002,"5min":388.96100000000001,"15min":388.96100000000001},"last":74.667000000000002},{"interface":"front","average":{"1min":16.219000000000001,"5min":32.975000000000001,"15min":35.768000000000001},"min":{"1min":8.5519999999999996,"5min":8.5519999999999996,"15min":8.5519999999999996},"max":{"1min":29.940999999999999,"5min":448.69200000000001,"15min":448.69200000000001},"last":35.981000000000002}]},{"osd":7,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":18.414999999999999,"5min":25.361000000000001,"15min":26.518999999999998},"min":{"1min":10.779,"5min":10.779,"15min":10.779},"max":{"1min":31.709,"5min":177.596,"15min":177.596},"last":70.296000000000006},{"interface":"front","average":{"1min":18.221,"5min":26.725000000000001,"15min":28.141999999999999},"min":{"1min":11.076000000000001,"5min":9.6270000000000007,"15min":9.6270000000000007},"max":{"1min":32.942999999999998,"5min":231.03800000000001,"15min":231.03800000000001},"last":76.353999999999999}]}]},{"osd":4,"up_from":20,"seq":85899345958,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":196896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:01:18 2024","interfaces":[{"interface":"back","average":{"1min":7.3579999999999997,"5min":11.317,"15min":11.977},"min":{"1min":4.7130000000000001,"5min":4.7130000000000001,"15min":4.7130000000000001},"max":{"1min":12.428000000000001,"5min":35.817999999999998,"15min":35.817999999999998},"last":35.052},{"interface":"front","average":{"1min":9.1029999999999998,"5min":11.619999999999999,"15min":12.039999999999999},"min":{"1min":5.9109999999999996,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":27.908000000000001,"5min":29.565999999999999,"15min":29.565999999999999},"last":29.084}]},{"osd":1,"last update":"Fri Aug 30 16:01:25 2024","interfaces":[{"interface":"back","average":{"1min":12.954000000000001,"5min":11.679,"15min":11.467000000000001},"min":{"1min":7.4450000000000003,"5min":4.7000000000000002,"15min":4.7000000000000002},"max":{"1min":28.349,"5min":28.423999999999999,"15min":28.423999999999999},"last":16.222000000000001},{"interface":"front","average":{"1min":11.127000000000001,"5min":12.247,"15min":12.433999999999999},"min":{"1min":7.2539999999999996,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":17.498000000000001,"5min":27.606999999999999,"15min":27.606999999999999},"last":17.597999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:01:18 2024","interfaces":[{"interface":"back","average":{"1min":9.9879999999999995,"5min":10.092000000000001,"15min":10.109999999999999},"min":{"1min":7.266,"5min":7.149,"15min":7.149},"max":{"1min":13.691000000000001,"5min":17.338000000000001,"15min":17.338000000000001},"last":5.4279999999999999},{"interface":"front","average":{"1min":13.154,"5min":14.414,"15min":14.624000000000001},"min":{"1min":9.6340000000000003,"5min":9.1539999999999999,"15min":9.1539999999999999},"max":{"1min":18.510000000000002,"5min":30.629000000000001,"15min":30.629000000000001},"last":14.948}]},{"osd":3,"last update":"Fri Aug 30 16:01:18 2024","interfaces":[{"interface":"back","average":{"1min":13.942,"5min":12.694000000000001,"15min":12.486000000000001},"min":{"1min":7.0259999999999998,"5min":7.0259999999999998,"15min":7.0259999999999998},"max":{"1min":29.715,"5min":29.715,"15min":29.715},"last":22.134},{"interface":"front","average":{"1min":13.914999999999999,"5min":15.358000000000001,"15min":15.598000000000001},"min":{"1min":9.3379999999999992,"5min":9.3379999999999992,"15min":9.3379999999999992},"max":{"1min":19.870999999999999,"5min":31.73,"15min":31.73},"last":21.065999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:01:18 2024","interfaces":[{"interface":"back","average":{"1min":12.464,"5min":11.422000000000001,"15min":11.247999999999999},"min":{"1min":8.423,"5min":6.5010000000000003,"15min":6.5010000000000003},"max":{"1min":24.498999999999999,"5min":27.966000000000001,"15min":27.966000000000001},"last":13.98},{"interface":"front","average":{"1min":12.878,"5min":14.144,"15min":14.355},"min":{"1min":8.548,"5min":8.548,"15min":8.548},"max":{"1min":19.527000000000001,"5min":24.367999999999999,"15min":24.367999999999999},"last":19.042999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:01:18 2024","interfaces":[{"interface":"back","average":{"1min":14.413,"5min":15.800000000000001,"15min":16.030999999999999},"min":{"1min":9.923,"5min":9.923,"15min":9.923},"max":{"1min":23.440000000000001,"5min":37.215000000000003,"15min":37.215000000000003},"last":33.448},{"interface":"front","average":{"1min":14.525,"5min":16.852,"15min":17.239999999999998},"min":{"1min":10.442,"5min":8.9719999999999995,"15min":8.9719999999999995},"max":{"1min":23.710999999999999,"5min":37.753999999999998,"15min":37.753999999999998},"last":19.542999999999999}]},{"osd":7,"last update":"Fri Aug 30 16:01:18 2024","interfaces":[{"interface":"back","average":{"1min":16.364000000000001,"5min":16.02,"15min":15.962},"min":{"1min":8.3960000000000008,"5min":8.3960000000000008,"15min":8.3960000000000008},"max":{"1min":30.341000000000001,"5min":34.551000000000002,"15min":34.551000000000002},"last":18.143000000000001},{"interface":"front","average":{"1min":18.567,"5min":17.776,"15min":17.643999999999998},"min":{"1min":12.385999999999999,"5min":12.385999999999999,"15min":12.385999999999999},"max":{"1min":31.736999999999998,"5min":31.736999999999998,"15min":31.736999999999998},"last":35.701999999999998}]}]},{"osd":2,"up_from":16,"seq":68719476772,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":438272,"data_stored":188300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:01:10 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":9.8279999999999994,"15min":9.7240000000000002},"min":{"1min":4.641,"5min":3.6709999999999998,"15min":3.6709999999999998},"max":{"1min":29.303000000000001,"5min":30.366,"15min":30.366},"last":12.912000000000001},{"interface":"front","average":{"1min":10.169,"5min":10.848000000000001,"15min":10.961},"min":{"1min":5.766,"5min":3.4900000000000002,"15min":3.4900000000000002},"max":{"1min":27.916,"5min":45.604999999999997,"15min":45.604999999999997},"last":15.587}]},{"osd":1,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":12.766999999999999,"5min":13.087999999999999,"15min":13.141999999999999},"min":{"1min":7.9829999999999997,"5min":7.6230000000000002,"15min":7.6230000000000002},"max":{"1min":25.483000000000001,"5min":46.460000000000001,"15min":46.460000000000001},"last":11.733000000000001},{"interface":"front","average":{"1min":11.375999999999999,"5min":12.794,"15min":13.029999999999999},"min":{"1min":5.944,"5min":5.944,"15min":5.944},"max":{"1min":21.670000000000002,"5min":42.561999999999998,"15min":42.561999999999998},"last":29.091000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":12.414999999999999,"5min":16.949000000000002,"15min":17.704999999999998},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":22.491,"5min":44.281999999999996,"15min":44.281999999999996},"last":18.199999999999999},{"interface":"front","average":{"1min":13.161,"5min":19.216999999999999,"15min":20.225999999999999},"min":{"1min":7.2590000000000003,"5min":7.2590000000000003,"15min":7.2590000000000003},"max":{"1min":34.459000000000003,"5min":50.564,"15min":50.564},"last":14.401999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":14.279,"5min":18.041,"15min":18.667999999999999},"min":{"1min":7.4900000000000002,"5min":7.4900000000000002,"15min":7.4900000000000002},"max":{"1min":33.892000000000003,"5min":48.673999999999999,"15min":48.673999999999999},"last":8.3539999999999992},{"interface":"front","average":{"1min":12.798,"5min":15.494,"15min":15.943},"min":{"1min":7.5529999999999999,"5min":6.9089999999999998,"15min":6.9089999999999998},"max":{"1min":23.937999999999999,"5min":40.823,"15min":40.823},"last":30.289999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:01:10 2024","interfaces":[{"interface":"back","average":{"1min":14.789,"5min":16.649999999999999,"15min":16.960000000000001},"min":{"1min":7.609,"5min":4.7199999999999998,"15min":4.7199999999999998},"max":{"1min":54.713000000000001,"5min":54.713000000000001,"15min":54.713000000000001},"last":31.158000000000001},{"interface":"front","average":{"1min":18.369,"5min":19.765999999999998,"15min":19.998999999999999},"min":{"1min":8.6910000000000007,"5min":5.7460000000000004,"15min":5.7460000000000004},"max":{"1min":56.152000000000001,"5min":56.917999999999999,"15min":56.917999999999999},"last":9.3339999999999996}]},{"osd":6,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":14.629,"5min":18.114000000000001,"15min":18.695},"min":{"1min":8.6359999999999992,"5min":8.6359999999999992,"15min":8.6359999999999992},"max":{"1min":27.744,"5min":72.75,"15min":72.75},"last":17.631},{"interface":"front","average":{"1min":13.335000000000001,"5min":19.451000000000001,"15min":20.469999999999999},"min":{"1min":8.4120000000000008,"5min":8.1579999999999995,"15min":8.1579999999999995},"max":{"1min":20.768999999999998,"5min":74.119,"15min":74.119},"last":40.896000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":24.567,"5min":22.645,"15min":22.324999999999999},"min":{"1min":11.906000000000001,"5min":11.906000000000001,"15min":11.906000000000001},"max":{"1min":137.33000000000001,"5min":137.33000000000001,"15min":137.33000000000001},"last":10.244},{"interface":"front","average":{"1min":22.824999999999999,"5min":20.588999999999999,"15min":20.216000000000001},"min":{"1min":10.179,"5min":10.179,"15min":10.179},"max":{"1min":136.66399999999999,"5min":136.66399999999999,"15min":136.66399999999999},"last":39.988999999999997}]}]},{"osd":0,"up_from":15,"seq":64424509475,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":192598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":10.805,"5min":9.891,"15min":9.7390000000000008},"min":{"1min":5.4080000000000004,"5min":5.2140000000000004,"15min":5.2140000000000004},"max":{"1min":23.588000000000001,"5min":23.588000000000001,"15min":23.588000000000001},"last":7.391},{"interface":"front","average":{"1min":10.755000000000001,"5min":11.234,"15min":11.314},"min":{"1min":6.0970000000000004,"5min":4.9000000000000004,"15min":4.9000000000000004},"max":{"1min":21.140000000000001,"5min":21.140000000000001,"15min":21.140000000000001},"last":9.4250000000000007}]},{"osd":2,"last update":"Fri Aug 30 16:01:09 2024","interfaces":[{"interface":"back","average":{"1min":12.598000000000001,"5min":11.487,"15min":11.302},"min":{"1min":6.1660000000000004,"5min":3.3370000000000002,"15min":3.3370000000000002},"max":{"1min":46.768999999999998,"5min":46.768999999999998,"15min":46.768999999999998},"last":12.584},{"interface":"front","average":{"1min":15.909000000000001,"5min":14.728,"15min":14.531000000000001},"min":{"1min":8.8580000000000005,"5min":5.702,"15min":5.702},"max":{"1min":43.921999999999997,"5min":43.921999999999997,"15min":43.921999999999997},"last":15.975}]},{"osd":3,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":16.379000000000001,"5min":23.527000000000001,"15min":24.719000000000001},"min":{"1min":7.069,"5min":7.069,"15min":7.069},"max":{"1min":38.219000000000001,"5min":200.155,"15min":200.155},"last":13.416},{"interface":"front","average":{"1min":20.651,"5min":29.088000000000001,"15min":30.494},"min":{"1min":10.94,"5min":7.8890000000000002,"15min":7.8890000000000002},"max":{"1min":44.229999999999997,"5min":267.03100000000001,"15min":267.03100000000001},"last":14.571}]},{"osd":4,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":15.583,"15min":15.863},"min":{"1min":6.6429999999999998,"5min":6.6429999999999998,"15min":6.6429999999999998},"max":{"1min":29.486000000000001,"5min":56.200000000000003,"15min":56.200000000000003},"last":16.510999999999999},{"interface":"front","average":{"1min":16.428999999999998,"5min":15.878,"15min":15.786},"min":{"1min":10.831,"5min":8.609,"15min":8.609},"max":{"1min":28.393999999999998,"5min":48.534999999999997,"15min":48.534999999999997},"last":13.946999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:01:09 2024","interfaces":[{"interface":"back","average":{"1min":16.562999999999999,"5min":16.338999999999999,"15min":16.302},"min":{"1min":10.868,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":31.952999999999999,"5min":31.952999999999999,"15min":31.952999999999999},"last":17.786999999999999},{"interface":"front","average":{"1min":17.684999999999999,"5min":17.885000000000002,"15min":17.917999999999999},"min":{"1min":9.2149999999999999,"5min":4.75,"15min":4.75},"max":{"1min":36.399000000000001,"5min":49.396000000000001,"15min":49.396000000000001},"last":15.141}]},{"osd":6,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":15.315,"5min":14.609999999999999,"15min":14.492000000000001},"min":{"1min":8.5380000000000003,"5min":7.6449999999999996,"15min":7.6449999999999996},"max":{"1min":31.059000000000001,"5min":34.569000000000003,"15min":34.569000000000003},"last":59.328000000000003},{"interface":"front","average":{"1min":18.312999999999999,"5min":18.571000000000002,"15min":18.614000000000001},"min":{"1min":11.446999999999999,"5min":11.446999999999999,"15min":11.446999999999999},"max":{"1min":32.743000000000002,"5min":55.103000000000002,"15min":55.103000000000002},"last":58.567}]},{"osd":7,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":19.716999999999999,"5min":21.596,"15min":21.908999999999999},"min":{"1min":11.154999999999999,"5min":9.1180000000000003,"15min":9.1180000000000003},"max":{"1min":39.518000000000001,"5min":59.515999999999998,"15min":59.515999999999998},"last":17.292999999999999},{"interface":"front","average":{"1min":14.148,"5min":15.448,"15min":15.664},"min":{"1min":7.1619999999999999,"5min":7.1619999999999999,"15min":7.1619999999999999},"max":{"1min":37.113,"5min":58.807000000000002,"15min":58.807000000000002},"last":21.451000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27524,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344316,"statfs":{"total":96636764160,"available":96608579584,"internally_reserved":0,"allocated":507904,"data_stored":603332,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:01:11 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":13.411,"15min":14.237},"min":{"1min":4.7400000000000002,"5min":4.5279999999999996,"15min":4.5279999999999996},"max":{"1min":17.573,"5min":111.313,"15min":111.313},"last":9.5660000000000007},{"interface":"front","average":{"1min":9.2080000000000002,"5min":14.778,"15min":15.706},"min":{"1min":5.1399999999999997,"5min":4.891,"15min":4.891},"max":{"1min":15.259,"5min":107.937,"15min":107.937},"last":7.4290000000000003}]},{"osd":1,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":10.409000000000001,"5min":12.935,"15min":13.356},"min":{"1min":6.0759999999999996,"5min":5.1539999999999999,"15min":5.1539999999999999},"max":{"1min":17.178999999999998,"5min":46.988,"15min":46.988},"last":8.5039999999999996},{"interface":"front","average":{"1min":14.106999999999999,"5min":15.887,"15min":16.183},"min":{"1min":8.5239999999999991,"5min":8.5239999999999991,"15min":8.5239999999999991},"max":{"1min":30.672999999999998,"5min":47.661000000000001,"15min":47.661000000000001},"last":10.465999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:01:11 2024","interfaces":[{"interface":"back","average":{"1min":16.370999999999999,"5min":18.73,"15min":19.123000000000001},"min":{"1min":6.9569999999999999,"5min":4.7220000000000004,"15min":4.7220000000000004},"max":{"1min":30.062000000000001,"5min":84.317999999999998,"15min":84.317999999999998},"last":11.531000000000001},{"interface":"front","average":{"1min":14.273,"5min":16.742000000000001,"15min":17.154},"min":{"1min":6.2770000000000001,"5min":6.0279999999999996,"15min":6.0279999999999996},"max":{"1min":25.507000000000001,"5min":99.884,"15min":99.884},"last":17.157}]},{"osd":3,"last update":"Fri Aug 30 16:01:19 2024","interfaces":[{"interface":"back","average":{"1min":23.041,"5min":18.285,"15min":17.492999999999999},"min":{"1min":8.907,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":117.346,"5min":117.346,"15min":117.346},"last":12.388},{"interface":"front","average":{"1min":21.218,"5min":17.446000000000002,"15min":16.817},"min":{"1min":8.1110000000000007,"5min":8.1110000000000007,"15min":8.1110000000000007},"max":{"1min":119.23099999999999,"5min":119.23099999999999,"15min":119.23099999999999},"last":18.872}]},{"osd":4,"last update":"Fri Aug 30 16:01:19 2024","interfaces":[{"interface":"back","average":{"1min":19.033000000000001,"5min":18.283999999999999,"15min":18.158999999999999},"min":{"1min":10.904,"5min":10.904,"15min":10.904},"max":{"1min":29.536999999999999,"5min":38.095999999999997,"15min":38.095999999999997},"last":32.887},{"interface":"front","average":{"1min":17.591999999999999,"5min":17.562000000000001,"15min":17.556999999999999},"min":{"1min":10.004,"5min":9.8450000000000006,"15min":9.8450000000000006},"max":{"1min":28.657,"5min":40.649000000000001,"15min":40.649000000000001},"last":30.841000000000001}]},{"osd":6,"last update":"Fri Aug 30 16:01:19 2024","interfaces":[{"interface":"back","average":{"1min":11.15,"5min":11.186,"15min":11.192},"min":{"1min":6.367,"5min":6.367,"15min":6.367},"max":{"1min":26.344999999999999,"5min":26.344999999999999,"15min":26.344999999999999},"last":18.053999999999998},{"interface":"front","average":{"1min":18.901,"5min":17.577000000000002,"15min":17.356000000000002},"min":{"1min":13.962,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":31.670000000000002,"5min":32.237000000000002,"15min":32.237000000000002},"last":13.800000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:01:19 2024","interfaces":[{"interface":"back","average":{"1min":15.496,"5min":17.890999999999998,"15min":18.289999999999999},"min":{"1min":9.0909999999999993,"5min":9.0909999999999993,"15min":9.0909999999999993},"max":{"1min":30.620999999999999,"5min":43.604999999999997,"15min":43.604999999999997},"last":12.997},{"interface":"front","average":{"1min":11.42,"5min":17.413,"15min":18.411999999999999},"min":{"1min":6.5570000000000004,"5min":6.5570000000000004,"15min":6.5570000000000004},"max":{"1min":19.026,"5min":88.278000000000006,"15min":88.278000000000006},"last":16.675999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:01:55.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:01:55.434 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:55.434 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:55.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:55.446 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:55.444+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:55.547 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:55.545+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:55.589 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:55.589 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:55.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:55.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-08-30T16:01:55.647 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:55.645+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:55.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:01:55.745 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:55.745 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:55.747 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:55.746+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:55.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:01:55.849 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:55.847+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:55.894 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:55.894 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:55.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:01:55.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:01:55.949 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:55.948+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:55.976 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:01:55.977 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:01:55.990 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-08-30T16:01:55.896456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":397859,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":192380,"kb_used_data":3216,"kb_used_omap":10,"kb_used_meta":189045,"kb_avail":660410500,"statfs":{"total":676457349120,"available":676260352000,"internally_reserved":0,"allocated":3293184,"data_stored":2191182,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":11157,"internal_metadata":193582187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":75,"apply_latency_ms":75,"commit_latency_ns":75000000,"apply_latency_ns":75000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.406049"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":45,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-08-30T16:01:52.857148+0000","last_change":"2024-08-30T15:59:27.646756+0000","last_active":"2024-08-30T16:01:52.857148+0000","last_peered":"2024-08-30T16:01:52.857148+0000","last_clean":"2024-08-30T16:01:52.857148+0000","last_became_active":"2024-08-30T15:59:25.342321+0000","last_became_peered":"2024-08-30T15:59:25.342321+0000","last_unstale":"2024-08-30T16:01:52.857148+0000","last_undegraded":"2024-08-30T16:01:52.857148+0000","last_fullsized":"2024-08-30T16:01:52.857148+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:53:42.927368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.116914037,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":43,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-08-30T16:01:50.101982+0000","last_change":"2024-08-30T15:59:28.156737+0000","last_active":"2024-08-30T16:01:50.101982+0000","last_peered":"2024-08-30T16:01:50.101982+0000","last_clean":"2024-08-30T16:01:50.101982+0000","last_became_active":"2024-08-30T15:59:25.662524+0000","last_became_peered":"2024-08-30T15:59:25.662524+0000","last_unstale":"2024-08-30T16:01:50.101982+0000","last_undegraded":"2024-08-30T16:01:50.101982+0000","last_fullsized":"2024-08-30T16:01:50.101982+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:05:30.092449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.064329163999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-08-30T16:01:52.851234+0000","last_change":"2024-08-30T15:59:27.633478+0000","last_active":"2024-08-30T16:01:52.851234+0000","last_peered":"2024-08-30T16:01:52.851234+0000","last_clean":"2024-08-30T16:01:52.851234+0000","last_became_active":"2024-08-30T15:59:24.572730+0000","last_became_peered":"2024-08-30T15:59:24.572730+0000","last_unstale":"2024-08-30T16:01:52.851234+0000","last_undegraded":"2024-08-30T16:01:52.851234+0000","last_fullsized":"2024-08-30T16:01:52.851234+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:36:20.950639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-08-30T16:01:53.103146+0000","last_change":"2024-08-30T16:01:00.718949+0000","last_active":"2024-08-30T16:01:53.103146+0000","last_peered":"2024-08-30T16:01:53.103146+0000","last_clean":"2024-08-30T16:01:53.103146+0000","last_became_active":"2024-08-30T16:01:00.713130+0000","last_became_peered":"2024-08-30T16:01:00.713130+0000","last_unstale":"2024-08-30T16:01:53.103146+0000","last_undegraded":"2024-08-30T16:01:53.103146+0000","last_fullsized":"2024-08-30T16:01:53.103146+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T03:38:51.475159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":59,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-08-30T16:01:52.838341+0000","last_change":"2024-08-30T16:01:01.435839+0000","last_active":"2024-08-30T16:01:52.838341+0000","last_peered":"2024-08-30T16:01:52.838341+0000","last_clean":"2024-08-30T16:01:52.838341+0000","last_became_active":"2024-08-30T16:01:00.974833+0000","last_became_peered":"2024-08-30T16:01:00.974833+0000","last_unstale":"2024-08-30T16:01:52.838341+0000","last_undegraded":"2024-08-30T16:01:52.838341+0000","last_fullsized":"2024-08-30T16:01:52.838341+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:09:38.575833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-08-30T16:01:52.832419+0000","last_change":"2024-08-30T15:59:27.825532+0000","last_active":"2024-08-30T16:01:52.832419+0000","last_peered":"2024-08-30T16:01:52.832419+0000","last_clean":"2024-08-30T16:01:52.832419+0000","last_became_active":"2024-08-30T15:59:25.469390+0000","last_became_peered":"2024-08-30T15:59:25.469390+0000","last_unstale":"2024-08-30T16:01:52.832419+0000","last_undegraded":"2024-08-30T16:01:52.832419+0000","last_fullsized":"2024-08-30T16:01:52.832419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T17:46:39.688911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":45,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-08-30T16:01:52.828583+0000","last_change":"2024-08-30T15:59:27.583986+0000","last_active":"2024-08-30T16:01:52.828583+0000","last_peered":"2024-08-30T16:01:52.828583+0000","last_clean":"2024-08-30T16:01:52.828583+0000","last_became_active":"2024-08-30T15:59:24.528536+0000","last_became_peered":"2024-08-30T15:59:24.528536+0000","last_unstale":"2024-08-30T16:01:52.828583+0000","last_undegraded":"2024-08-30T16:01:52.828583+0000","last_fullsized":"2024-08-30T16:01:52.828583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:18:40.602710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-08-30T16:01:52.842944+0000","last_change":"2024-08-30T16:01:01.316424+0000","last_active":"2024-08-30T16:01:52.842944+0000","last_peered":"2024-08-30T16:01:52.842944+0000","last_clean":"2024-08-30T16:01:52.842944+0000","last_became_active":"2024-08-30T16:01:00.806973+0000","last_became_peered":"2024-08-30T16:01:00.806973+0000","last_unstale":"2024-08-30T16:01:52.842944+0000","last_undegraded":"2024-08-30T16:01:52.842944+0000","last_fullsized":"2024-08-30T16:01:52.842944+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T23:28:24.797655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":47,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-08-30T16:01:53.639077+0000","last_change":"2024-08-30T15:59:21.623783+0000","last_active":"2024-08-30T16:01:53.639077+0000","last_peered":"2024-08-30T16:01:53.639077+0000","last_clean":"2024-08-30T16:01:53.639077+0000","last_became_active":"2024-08-30T15:59:20.848924+0000","last_became_peered":"2024-08-30T15:59:20.848924+0000","last_unstale":"2024-08-30T16:01:53.639077+0000","last_undegraded":"2024-08-30T16:01:53.639077+0000","last_fullsized":"2024-08-30T16:01:53.639077+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:14:52.575421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":795680,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27524,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344316,"statfs":{"total":96636764160,"available":96608579584,"internally_reserved":0,"allocated":516096,"data_stored":607649,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":10.308999999999999,"5min":13.598000000000001,"15min":14.146000000000001},"min":{"1min":5.585,"5min":5.585,"15min":5.585},"max":{"1min":23.088000000000001,"5min":57.177999999999997,"15min":57.177999999999997},"last":8.4649999999999999},{"interface":"front","average":{"1min":11.601000000000001,"5min":12.992000000000001,"15min":13.224},"min":{"1min":5.0510000000000002,"5min":5.0510000000000002,"15min":5.0510000000000002},"max":{"1min":29.195,"5min":58.024999999999999,"15min":58.024999999999999},"last":15.694000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":11.406000000000001,"5min":10.624000000000001,"15min":10.494},"min":{"1min":6.117,"5min":6.117,"15min":6.117},"max":{"1min":73.399000000000001,"5min":73.399000000000001,"15min":73.399000000000001},"last":17.044},{"interface":"front","average":{"1min":16.138000000000002,"5min":15.289999999999999,"15min":15.148},"min":{"1min":9.7469999999999999,"5min":6.9139999999999997,"15min":6.9139999999999997},"max":{"1min":75.120000000000005,"5min":75.120000000000005,"15min":75.120000000000005},"last":11.824999999999999}]},{"osd":3,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":14.863,"5min":17.103000000000002,"15min":17.477},"min":{"1min":7.0410000000000004,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":21.734000000000002,"5min":43.067999999999998,"15min":43.067999999999998},"last":19.641999999999999},{"interface":"front","average":{"1min":17.216000000000001,"5min":15.385,"15min":15.08},"min":{"1min":7.6619999999999999,"5min":7.6619999999999999,"15min":7.6619999999999999},"max":{"1min":104.38200000000001,"5min":104.38200000000001,"15min":104.38200000000001},"last":20.210000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":15.086,"5min":14.839,"15min":14.798},"min":{"1min":7.6950000000000003,"5min":7.6950000000000003,"15min":7.6950000000000003},"max":{"1min":30.754999999999999,"5min":30.754999999999999,"15min":30.754999999999999},"last":9.7810000000000006},{"interface":"front","average":{"1min":13.462999999999999,"5min":13.144,"15min":13.090999999999999},"min":{"1min":7.0039999999999996,"5min":7.0039999999999996,"15min":7.0039999999999996},"max":{"1min":29.818999999999999,"5min":29.818999999999999,"15min":29.818999999999999},"last":13.375999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":12.845000000000001,"5min":16.100999999999999,"15min":16.643000000000001},"min":{"1min":8.2170000000000005,"5min":8.1950000000000003,"15min":8.1950000000000003},"max":{"1min":29.474,"5min":46.100000000000001,"15min":46.100000000000001},"last":38.088000000000001},{"interface":"front","average":{"1min":17.539999999999999,"5min":19.335999999999999,"15min":19.635999999999999},"min":{"1min":9.8179999999999996,"5min":9.8179999999999996,"15min":9.8179999999999996},"max":{"1min":31.367000000000001,"5min":45.216999999999999,"15min":45.216999999999999},"last":39.103999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":17.210999999999999,"5min":17.657,"15min":17.731000000000002},"min":{"1min":9.1929999999999996,"5min":9.1929999999999996,"15min":9.1929999999999996},"max":{"1min":36.987000000000002,"5min":58.201999999999998,"15min":58.201999999999998},"last":12.574999999999999},{"interface":"front","average":{"1min":17.010000000000002,"5min":17.103999999999999,"15min":17.120000000000001},"min":{"1min":9.8279999999999994,"5min":8.8390000000000004,"15min":8.8390000000000004},"max":{"1min":35.582999999999998,"5min":37.253,"15min":37.253},"last":18.946999999999999}]},{"osd":7,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":17.559000000000001,"5min":19.727,"15min":20.088999999999999},"min":{"1min":12.42,"5min":11.590999999999999,"15min":11.590999999999999},"max":{"1min":29.251999999999999,"5min":40.990000000000002,"15min":40.990000000000002},"last":15.007999999999999},{"interface":"front","average":{"1min":17.387,"5min":19.103000000000002,"15min":19.388999999999999},"min":{"1min":8.3640000000000008,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":32.363,"5min":59.079000000000001,"15min":59.079000000000001},"last":16.234000000000002}]}]},{"osd":7,"up_from":21,"seq":90194313255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":201194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":8.8529999999999998,"5min":8.9410000000000007,"15min":8.9559999999999995},"min":{"1min":4.899,"5min":4.7169999999999996,"15min":4.7169999999999996},"max":{"1min":17.946000000000002,"5min":30.600999999999999,"15min":30.600999999999999},"last":25.759},{"interface":"front","average":{"1min":8.7629999999999999,"5min":9.0510000000000002,"15min":9.0990000000000002},"min":{"1min":5.1440000000000001,"5min":5.0659999999999998,"15min":5.0659999999999998},"max":{"1min":21.960999999999999,"5min":21.960999999999999,"15min":21.960999999999999},"last":28.077000000000002}]},{"osd":1,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":16.844000000000001,"5min":12.037000000000001,"15min":11.236000000000001},"min":{"1min":7.6989999999999998,"5min":6.4180000000000001,"15min":6.4180000000000001},"max":{"1min":83.581000000000003,"5min":83.581000000000003,"15min":83.581000000000003},"last":5.7990000000000004},{"interface":"front","average":{"1min":14.843999999999999,"5min":12.894,"15min":12.569000000000001},"min":{"1min":7.7380000000000004,"5min":7.2590000000000003,"15min":7.2590000000000003},"max":{"1min":81.722999999999999,"5min":81.722999999999999,"15min":81.722999999999999},"last":8.8309999999999995}]},{"osd":2,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":14.759,"5min":13.823,"15min":13.667},"min":{"1min":9.6859999999999999,"5min":6.867,"15min":6.867},"max":{"1min":29.75,"5min":29.75,"15min":29.75},"last":29.251000000000001},{"interface":"front","average":{"1min":13.172000000000001,"5min":16.649000000000001,"15min":17.228999999999999},"min":{"1min":6.9800000000000004,"5min":6.9800000000000004,"15min":6.9800000000000004},"max":{"1min":30.853999999999999,"5min":36.134999999999998,"15min":36.134999999999998},"last":12.957000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":20.297000000000001,"5min":17.335999999999999,"15min":16.841999999999999},"min":{"1min":9.1980000000000004,"5min":8.8420000000000005,"15min":8.8420000000000005},"max":{"1min":65.739999999999995,"5min":65.739999999999995,"15min":65.739999999999995},"last":10.497999999999999},{"interface":"front","average":{"1min":18.071000000000002,"5min":13.746,"15min":13.025},"min":{"1min":7.4020000000000001,"5min":7.3849999999999998,"15min":7.3849999999999998},"max":{"1min":64.980000000000004,"5min":64.980000000000004,"15min":64.980000000000004},"last":9.6120000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":15.869999999999999,"5min":13.037000000000001,"15min":12.565},"min":{"1min":7.6710000000000003,"5min":7.0339999999999998,"15min":7.0339999999999998},"max":{"1min":37.746000000000002,"5min":37.746000000000002,"15min":37.746000000000002},"last":12.167},{"interface":"front","average":{"1min":16.552,"5min":17.541,"15min":17.706},"min":{"1min":11.145,"5min":9.2070000000000007,"15min":9.2070000000000007},"max":{"1min":32.526000000000003,"5min":37.393000000000001,"15min":37.393000000000001},"last":13.484}]},{"osd":5,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":12.071,"5min":11.478,"15min":11.379},"min":{"1min":6.2309999999999999,"5min":6.2309999999999999,"15min":6.2309999999999999},"max":{"1min":29.754999999999999,"5min":29.754999999999999,"15min":29.754999999999999},"last":7.3150000000000004},{"interface":"front","average":{"1min":16.251999999999999,"5min":16.863,"15min":16.965},"min":{"1min":10.015000000000001,"5min":9.8620000000000001,"15min":9.8620000000000001},"max":{"1min":38.618000000000002,"5min":38.618000000000002,"15min":38.618000000000002},"last":23.48}]},{"osd":6,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":16.870000000000001,"5min":13.715,"15min":13.189},"min":{"1min":8.9380000000000006,"5min":6.3230000000000004,"15min":6.3230000000000004},"max":{"1min":42.829999999999998,"5min":42.829999999999998,"15min":42.829999999999998},"last":11.244},{"interface":"front","average":{"1min":19.25,"5min":18.513999999999999,"15min":18.391999999999999},"min":{"1min":10.749000000000001,"5min":10.749000000000001,"15min":10.749000000000001},"max":{"1min":44.491999999999997,"5min":44.491999999999997,"15min":44.491999999999997},"last":25.225999999999999}]}]},{"osd":6,"up_from":0,"seq":90194313254,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":21,"seq":90194313254,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":201213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":8.3100000000000005,"5min":10.964,"15min":11.406000000000001},"min":{"1min":5.0890000000000004,"5min":4.29,"15min":4.29},"max":{"1min":13.206,"5min":50.500999999999998,"15min":50.500999999999998},"last":16.832000000000001},{"interface":"front","average":{"1min":9.8729999999999993,"5min":13.948,"15min":14.627000000000001},"min":{"1min":6.2069999999999999,"5min":6.2069999999999999,"15min":6.2069999999999999},"max":{"1min":25.311,"5min":51.965000000000003,"15min":51.965000000000003},"last":17.637}]},{"osd":1,"last update":"Fri Aug 30 16:01:22 2024","interfaces":[{"interface":"back","average":{"1min":14.113,"5min":18.672999999999998,"15min":19.433},"min":{"1min":8.2460000000000004,"5min":6.0229999999999997,"15min":6.0229999999999997},"max":{"1min":32.478999999999999,"5min":137.56700000000001,"15min":137.56700000000001},"last":19.919},{"interface":"front","average":{"1min":12.951000000000001,"5min":16.577999999999999,"15min":17.181999999999999},"min":{"1min":5.0060000000000002,"5min":5.0060000000000002,"15min":5.0060000000000002},"max":{"1min":23.946999999999999,"5min":136.321,"15min":136.321},"last":19.338999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":12.608000000000001,"5min":16.045999999999999,"15min":16.619},"min":{"1min":7.4219999999999997,"5min":7.4219999999999997,"15min":7.4219999999999997},"max":{"1min":25.902000000000001,"5min":73.831000000000003,"15min":73.831000000000003},"last":43.869999999999997},{"interface":"front","average":{"1min":12.295999999999999,"5min":15.694000000000001,"15min":16.260000000000002},"min":{"1min":7.5670000000000002,"5min":7.4169999999999998,"15min":7.4169999999999998},"max":{"1min":20.948,"5min":71.831999999999994,"15min":71.831999999999994},"last":8.1069999999999993}]},{"osd":4,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":15.612,"5min":28.602,"15min":30.766999999999999},"min":{"1min":8.2460000000000004,"5min":8.2460000000000004,"15min":8.2460000000000004},"max":{"1min":29.398,"5min":342.34500000000003,"15min":342.34500000000003},"last":43.235999999999997},{"interface":"front","average":{"1min":14.583,"5min":19.834,"15min":20.709},"min":{"1min":8.0299999999999994,"5min":8.0299999999999994,"15min":8.0299999999999994},"max":{"1min":28.853000000000002,"5min":127.857,"15min":127.857},"last":63.633000000000003}]},{"osd":5,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":13.646000000000001,"5min":15.612,"15min":15.94},"min":{"1min":7.2389999999999999,"5min":7.2389999999999999,"15min":7.2389999999999999},"max":{"1min":26.850999999999999,"5min":29.577999999999999,"15min":29.577999999999999},"last":33.68},{"interface":"front","average":{"1min":16.716999999999999,"5min":18.940999999999999,"15min":19.312000000000001},"min":{"1min":10.073,"5min":7.2370000000000001,"15min":7.2370000000000001},"max":{"1min":34.149000000000001,"5min":39.939,"15min":39.939},"last":58.212000000000003}]},{"osd":6,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":16.007999999999999,"5min":29.869,"15min":32.179000000000002},"min":{"1min":11.974,"5min":8.6630000000000003,"15min":8.6630000000000003},"max":{"1min":33.237000000000002,"5min":388.96100000000001,"15min":388.96100000000001},"last":74.667000000000002},{"interface":"front","average":{"1min":16.219000000000001,"5min":32.975000000000001,"15min":35.768000000000001},"min":{"1min":8.5519999999999996,"5min":8.5519999999999996,"15min":8.5519999999999996},"max":{"1min":29.940999999999999,"5min":448.69200000000001,"15min":448.69200000000001},"last":35.981000000000002}]},{"osd":7,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":18.414999999999999,"5min":25.361000000000001,"15min":26.518999999999998},"min":{"1min":10.779,"5min":10.779,"15min":10.779},"max":{"1min":31.709,"5min":177.596,"15min":177.596},"last":70.296000000000006},{"interface":"front","average":{"1min":18.221,"5min":26.725000000000001,"15min":28.141999999999999},"min":{"1min":11.076000000000001,"5min":9.6270000000000007,"15min":9.6270000000000007},"max":{"1min":32.942999999999998,"5min":231.03800000000001,"15min":231.03800000000001},"last":76.353999999999999}]}]},{"osd":4,"up_from":20,"seq":85899345958,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":196896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:01:18 2024","interfaces":[{"interface":"back","average":{"1min":7.3579999999999997,"5min":11.317,"15min":11.977},"min":{"1min":4.7130000000000001,"5min":4.7130000000000001,"15min":4.7130000000000001},"max":{"1min":12.428000000000001,"5min":35.817999999999998,"15min":35.817999999999998},"last":35.052},{"interface":"front","average":{"1min":9.1029999999999998,"5min":11.619999999999999,"15min":12.039999999999999},"min":{"1min":5.9109999999999996,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":27.908000000000001,"5min":29.565999999999999,"15min":29.565999999999999},"last":29.084}]},{"osd":1,"last update":"Fri Aug 30 16:01:25 2024","interfaces":[{"interface":"back","average":{"1min":12.954000000000001,"5min":11.679,"15min":11.467000000000001},"min":{"1min":7.4450000000000003,"5min":4.7000000000000002,"15min":4.7000000000000002},"max":{"1min":28.349,"5min":28.423999999999999,"15min":28.423999999999999},"last":16.222000000000001},{"interface":"front","average":{"1min":11.127000000000001,"5min":12.247,"15min":12.433999999999999},"min":{"1min":7.2539999999999996,"5min":6.8520000000000003,"15min":6.8520000000000003},"max":{"1min":17.498000000000001,"5min":27.606999999999999,"15min":27.606999999999999},"last":17.597999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:01:18 2024","interfaces":[{"interface":"back","average":{"1min":9.9879999999999995,"5min":10.092000000000001,"15min":10.109999999999999},"min":{"1min":7.266,"5min":7.149,"15min":7.149},"max":{"1min":13.691000000000001,"5min":17.338000000000001,"15min":17.338000000000001},"last":5.4279999999999999},{"interface":"front","average":{"1min":13.154,"5min":14.414,"15min":14.624000000000001},"min":{"1min":9.6340000000000003,"5min":9.1539999999999999,"15min":9.1539999999999999},"max":{"1min":18.510000000000002,"5min":30.629000000000001,"15min":30.629000000000001},"last":14.948}]},{"osd":3,"last update":"Fri Aug 30 16:01:18 2024","interfaces":[{"interface":"back","average":{"1min":13.942,"5min":12.694000000000001,"15min":12.486000000000001},"min":{"1min":7.0259999999999998,"5min":7.0259999999999998,"15min":7.0259999999999998},"max":{"1min":29.715,"5min":29.715,"15min":29.715},"last":22.134},{"interface":"front","average":{"1min":13.914999999999999,"5min":15.358000000000001,"15min":15.598000000000001},"min":{"1min":9.3379999999999992,"5min":9.3379999999999992,"15min":9.3379999999999992},"max":{"1min":19.870999999999999,"5min":31.73,"15min":31.73},"last":21.065999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:01:18 2024","interfaces":[{"interface":"back","average":{"1min":12.464,"5min":11.422000000000001,"15min":11.247999999999999},"min":{"1min":8.423,"5min":6.5010000000000003,"15min":6.5010000000000003},"max":{"1min":24.498999999999999,"5min":27.966000000000001,"15min":27.966000000000001},"last":13.98},{"interface":"front","average":{"1min":12.878,"5min":14.144,"15min":14.355},"min":{"1min":8.548,"5min":8.548,"15min":8.548},"max":{"1min":19.527000000000001,"5min":24.367999999999999,"15min":24.367999999999999},"last":19.042999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:01:18 2024","interfaces":[{"interface":"back","average":{"1min":14.413,"5min":15.800000000000001,"15min":16.030999999999999},"min":{"1min":9.923,"5min":9.923,"15min":9.923},"max":{"1min":23.440000000000001,"5min":37.215000000000003,"15min":37.215000000000003},"last":33.448},{"interface":"front","average":{"1min":14.525,"5min":16.852,"15min":17.239999999999998},"min":{"1min":10.442,"5min":8.9719999999999995,"15min":8.9719999999999995},"max":{"1min":23.710999999999999,"5min":37.753999999999998,"15min":37.753999999999998},"last":19.542999999999999}]},{"osd":7,"last update":"Fri Aug 30 16:01:18 2024","interfaces":[{"interface":"back","average":{"1min":16.364000000000001,"5min":16.02,"15min":15.962},"min":{"1min":8.3960000000000008,"5min":8.3960000000000008,"15min":8.3960000000000008},"max":{"1min":30.341000000000001,"5min":34.551000000000002,"15min":34.551000000000002},"last":18.143000000000001},{"interface":"front","average":{"1min":18.567,"5min":17.776,"15min":17.643999999999998},"min":{"1min":12.385999999999999,"5min":12.385999999999999,"15min":12.385999999999999},"max":{"1min":31.736999999999998,"5min":31.736999999999998,"15min":31.736999999999998},"last":35.701999999999998}]}]},{"osd":2,"up_from":16,"seq":68719476772,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":438272,"data_stored":188300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:01:10 2024","interfaces":[{"interface":"back","average":{"1min":10.452999999999999,"5min":9.8279999999999994,"15min":9.7240000000000002},"min":{"1min":4.641,"5min":3.6709999999999998,"15min":3.6709999999999998},"max":{"1min":29.303000000000001,"5min":30.366,"15min":30.366},"last":12.912000000000001},{"interface":"front","average":{"1min":10.169,"5min":10.848000000000001,"15min":10.961},"min":{"1min":5.766,"5min":3.4900000000000002,"15min":3.4900000000000002},"max":{"1min":27.916,"5min":45.604999999999997,"15min":45.604999999999997},"last":15.587}]},{"osd":1,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":12.766999999999999,"5min":13.087999999999999,"15min":13.141999999999999},"min":{"1min":7.9829999999999997,"5min":7.6230000000000002,"15min":7.6230000000000002},"max":{"1min":25.483000000000001,"5min":46.460000000000001,"15min":46.460000000000001},"last":11.733000000000001},{"interface":"front","average":{"1min":11.375999999999999,"5min":12.794,"15min":13.029999999999999},"min":{"1min":5.944,"5min":5.944,"15min":5.944},"max":{"1min":21.670000000000002,"5min":42.561999999999998,"15min":42.561999999999998},"last":29.091000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":12.414999999999999,"5min":16.949000000000002,"15min":17.704999999999998},"min":{"1min":6.75,"5min":6.75,"15min":6.75},"max":{"1min":22.491,"5min":44.281999999999996,"15min":44.281999999999996},"last":18.199999999999999},{"interface":"front","average":{"1min":13.161,"5min":19.216999999999999,"15min":20.225999999999999},"min":{"1min":7.2590000000000003,"5min":7.2590000000000003,"15min":7.2590000000000003},"max":{"1min":34.459000000000003,"5min":50.564,"15min":50.564},"last":14.401999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":14.279,"5min":18.041,"15min":18.667999999999999},"min":{"1min":7.4900000000000002,"5min":7.4900000000000002,"15min":7.4900000000000002},"max":{"1min":33.892000000000003,"5min":48.673999999999999,"15min":48.673999999999999},"last":8.3539999999999992},{"interface":"front","average":{"1min":12.798,"5min":15.494,"15min":15.943},"min":{"1min":7.5529999999999999,"5min":6.9089999999999998,"15min":6.9089999999999998},"max":{"1min":23.937999999999999,"5min":40.823,"15min":40.823},"last":30.289999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:01:10 2024","interfaces":[{"interface":"back","average":{"1min":14.789,"5min":16.649999999999999,"15min":16.960000000000001},"min":{"1min":7.609,"5min":4.7199999999999998,"15min":4.7199999999999998},"max":{"1min":54.713000000000001,"5min":54.713000000000001,"15min":54.713000000000001},"last":31.158000000000001},{"interface":"front","average":{"1min":18.369,"5min":19.765999999999998,"15min":19.998999999999999},"min":{"1min":8.6910000000000007,"5min":5.7460000000000004,"15min":5.7460000000000004},"max":{"1min":56.152000000000001,"5min":56.917999999999999,"15min":56.917999999999999},"last":9.3339999999999996}]},{"osd":6,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":14.629,"5min":18.114000000000001,"15min":18.695},"min":{"1min":8.6359999999999992,"5min":8.6359999999999992,"15min":8.6359999999999992},"max":{"1min":27.744,"5min":72.75,"15min":72.75},"last":17.631},{"interface":"front","average":{"1min":13.335000000000001,"5min":19.451000000000001,"15min":20.469999999999999},"min":{"1min":8.4120000000000008,"5min":8.1579999999999995,"15min":8.1579999999999995},"max":{"1min":20.768999999999998,"5min":74.119,"15min":74.119},"last":40.896000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:01:21 2024","interfaces":[{"interface":"back","average":{"1min":24.567,"5min":22.645,"15min":22.324999999999999},"min":{"1min":11.906000000000001,"5min":11.906000000000001,"15min":11.906000000000001},"max":{"1min":137.33000000000001,"5min":137.33000000000001,"15min":137.33000000000001},"last":10.244},{"interface":"front","average":{"1min":22.824999999999999,"5min":20.588999999999999,"15min":20.216000000000001},"min":{"1min":10.179,"5min":10.179,"15min":10.179},"max":{"1min":136.66399999999999,"5min":136.66399999999999,"15min":136.66399999999999},"last":39.988999999999997}]}]},{"osd":0,"up_from":15,"seq":64424509475,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":192598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":10.805,"5min":9.891,"15min":9.7390000000000008},"min":{"1min":5.4080000000000004,"5min":5.2140000000000004,"15min":5.2140000000000004},"max":{"1min":23.588000000000001,"5min":23.588000000000001,"15min":23.588000000000001},"last":7.391},{"interface":"front","average":{"1min":10.755000000000001,"5min":11.234,"15min":11.314},"min":{"1min":6.0970000000000004,"5min":4.9000000000000004,"15min":4.9000000000000004},"max":{"1min":21.140000000000001,"5min":21.140000000000001,"15min":21.140000000000001},"last":9.4250000000000007}]},{"osd":2,"last update":"Fri Aug 30 16:01:09 2024","interfaces":[{"interface":"back","average":{"1min":12.598000000000001,"5min":11.487,"15min":11.302},"min":{"1min":6.1660000000000004,"5min":3.3370000000000002,"15min":3.3370000000000002},"max":{"1min":46.768999999999998,"5min":46.768999999999998,"15min":46.768999999999998},"last":12.584},{"interface":"front","average":{"1min":15.909000000000001,"5min":14.728,"15min":14.531000000000001},"min":{"1min":8.8580000000000005,"5min":5.702,"15min":5.702},"max":{"1min":43.921999999999997,"5min":43.921999999999997,"15min":43.921999999999997},"last":15.975}]},{"osd":3,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":16.379000000000001,"5min":23.527000000000001,"15min":24.719000000000001},"min":{"1min":7.069,"5min":7.069,"15min":7.069},"max":{"1min":38.219000000000001,"5min":200.155,"15min":200.155},"last":13.416},{"interface":"front","average":{"1min":20.651,"5min":29.088000000000001,"15min":30.494},"min":{"1min":10.94,"5min":7.8890000000000002,"15min":7.8890000000000002},"max":{"1min":44.229999999999997,"5min":267.03100000000001,"15min":267.03100000000001},"last":14.571}]},{"osd":4,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":15.583,"15min":15.863},"min":{"1min":6.6429999999999998,"5min":6.6429999999999998,"15min":6.6429999999999998},"max":{"1min":29.486000000000001,"5min":56.200000000000003,"15min":56.200000000000003},"last":16.510999999999999},{"interface":"front","average":{"1min":16.428999999999998,"5min":15.878,"15min":15.786},"min":{"1min":10.831,"5min":8.609,"15min":8.609},"max":{"1min":28.393999999999998,"5min":48.534999999999997,"15min":48.534999999999997},"last":13.946999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:01:09 2024","interfaces":[{"interface":"back","average":{"1min":16.562999999999999,"5min":16.338999999999999,"15min":16.302},"min":{"1min":10.868,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":31.952999999999999,"5min":31.952999999999999,"15min":31.952999999999999},"last":17.786999999999999},{"interface":"front","average":{"1min":17.684999999999999,"5min":17.885000000000002,"15min":17.917999999999999},"min":{"1min":9.2149999999999999,"5min":4.75,"15min":4.75},"max":{"1min":36.399000000000001,"5min":49.396000000000001,"15min":49.396000000000001},"last":15.141}]},{"osd":6,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":15.315,"5min":14.609999999999999,"15min":14.492000000000001},"min":{"1min":8.5380000000000003,"5min":7.6449999999999996,"15min":7.6449999999999996},"max":{"1min":31.059000000000001,"5min":34.569000000000003,"15min":34.569000000000003},"last":59.328000000000003},{"interface":"front","average":{"1min":18.312999999999999,"5min":18.571000000000002,"15min":18.614000000000001},"min":{"1min":11.446999999999999,"5min":11.446999999999999,"15min":11.446999999999999},"max":{"1min":32.743000000000002,"5min":55.103000000000002,"15min":55.103000000000002},"last":58.567}]},{"osd":7,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":19.716999999999999,"5min":21.596,"15min":21.908999999999999},"min":{"1min":11.154999999999999,"5min":9.1180000000000003,"15min":9.1180000000000003},"max":{"1min":39.518000000000001,"5min":59.515999999999998,"15min":59.515999999999998},"last":17.292999999999999},{"interface":"front","average":{"1min":14.148,"5min":15.448,"15min":15.664},"min":{"1min":7.1619999999999999,"5min":7.1619999999999999,"15min":7.1619999999999999},"max":{"1min":37.113,"5min":58.807000000000002,"15min":58.807000000000002},"last":21.451000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27524,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344316,"statfs":{"total":96636764160,"available":96608579584,"internally_reserved":0,"allocated":507904,"data_stored":603332,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:01:11 2024","interfaces":[{"interface":"back","average":{"1min":8.4559999999999995,"5min":13.411,"15min":14.237},"min":{"1min":4.7400000000000002,"5min":4.5279999999999996,"15min":4.5279999999999996},"max":{"1min":17.573,"5min":111.313,"15min":111.313},"last":9.5660000000000007},{"interface":"front","average":{"1min":9.2080000000000002,"5min":14.778,"15min":15.706},"min":{"1min":5.1399999999999997,"5min":4.891,"15min":4.891},"max":{"1min":15.259,"5min":107.937,"15min":107.937},"last":7.4290000000000003}]},{"osd":1,"last update":"Fri Aug 30 16:01:27 2024","interfaces":[{"interface":"back","average":{"1min":10.409000000000001,"5min":12.935,"15min":13.356},"min":{"1min":6.0759999999999996,"5min":5.1539999999999999,"15min":5.1539999999999999},"max":{"1min":17.178999999999998,"5min":46.988,"15min":46.988},"last":8.5039999999999996},{"interface":"front","average":{"1min":14.106999999999999,"5min":15.887,"15min":16.183},"min":{"1min":8.5239999999999991,"5min":8.5239999999999991,"15min":8.5239999999999991},"max":{"1min":30.672999999999998,"5min":47.661000000000001,"15min":47.661000000000001},"last":10.465999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:01:11 2024","interfaces":[{"interface":"back","average":{"1min":16.370999999999999,"5min":18.73,"15min":19.123000000000001},"min":{"1min":6.9569999999999999,"5min":4.7220000000000004,"15min":4.7220000000000004},"max":{"1min":30.062000000000001,"5min":84.317999999999998,"15min":84.317999999999998},"last":11.531000000000001},{"interface":"front","average":{"1min":14.273,"5min":16.742000000000001,"15min":17.154},"min":{"1min":6.2770000000000001,"5min":6.0279999999999996,"15min":6.0279999999999996},"max":{"1min":25.507000000000001,"5min":99.884,"15min":99.884},"last":17.157}]},{"osd":3,"last update":"Fri Aug 30 16:01:19 2024","interfaces":[{"interface":"back","average":{"1min":23.041,"5min":18.285,"15min":17.492999999999999},"min":{"1min":8.907,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":117.346,"5min":117.346,"15min":117.346},"last":12.388},{"interface":"front","average":{"1min":21.218,"5min":17.446000000000002,"15min":16.817},"min":{"1min":8.1110000000000007,"5min":8.1110000000000007,"15min":8.1110000000000007},"max":{"1min":119.23099999999999,"5min":119.23099999999999,"15min":119.23099999999999},"last":18.872}]},{"osd":4,"last update":"Fri Aug 30 16:01:19 2024","interfaces":[{"interface":"back","average":{"1min":19.033000000000001,"5min":18.283999999999999,"15min":18.158999999999999},"min":{"1min":10.904,"5min":10.904,"15min":10.904},"max":{"1min":29.536999999999999,"5min":38.095999999999997,"15min":38.095999999999997},"last":32.887},{"interface":"front","average":{"1min":17.591999999999999,"5min":17.562000000000001,"15min":17.556999999999999},"min":{"1min":10.004,"5min":9.8450000000000006,"15min":9.8450000000000006},"max":{"1min":28.657,"5min":40.649000000000001,"15min":40.649000000000001},"last":30.841000000000001}]},{"osd":6,"last update":"Fri Aug 30 16:01:19 2024","interfaces":[{"interface":"back","average":{"1min":11.15,"5min":11.186,"15min":11.192},"min":{"1min":6.367,"5min":6.367,"15min":6.367},"max":{"1min":26.344999999999999,"5min":26.344999999999999,"15min":26.344999999999999},"last":18.053999999999998},{"interface":"front","average":{"1min":18.901,"5min":17.577000000000002,"15min":17.356000000000002},"min":{"1min":13.962,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":31.670000000000002,"5min":32.237000000000002,"15min":32.237000000000002},"last":13.800000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:01:19 2024","interfaces":[{"interface":"back","average":{"1min":15.496,"5min":17.890999999999998,"15min":18.289999999999999},"min":{"1min":9.0909999999999993,"5min":9.0909999999999993,"15min":9.0909999999999993},"max":{"1min":30.620999999999999,"5min":43.604999999999997,"15min":43.604999999999997},"last":12.997},{"interface":"front","average":{"1min":11.42,"5min":17.413,"15min":18.411999999999999},"min":{"1min":6.5570000000000004,"5min":6.5570000000000004,"15min":6.5570000000000004},"max":{"1min":19.026,"5min":88.278000000000006,"15min":88.278000000000006},"last":16.675999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:01:55.992 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-08-30T16:01:55.992 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:01:55.992 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-08-30T16:01:55.992 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T16:01:55.992 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T16:01:55.993 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2024-08-30T16:01:56.048 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:56.048 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:56.050 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:56.049+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:56.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:01:56.124 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:56.124 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-30T16:01:56.125 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:56.151 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:56.149+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:56.252 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:56.250+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:56.280 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:01:56.280 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:01:56.280 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:56.280 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:56.293 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:01:56.347 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:08.073 40108== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:56.352 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:56.350+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:56.383 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:01:56.400 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:01:56.401 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:01:56.453 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:56.452+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:56.490 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:56.490 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:56.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:56.490 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:56.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.490 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:56.490 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:56.490 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 155.20990740100001, 2024-08-30T16:01:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:01:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:01:56.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:01:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:01:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:01:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:01:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:01:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:01:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:01:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:01:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:01:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:01:56.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:01:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:01:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:01:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:01:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:56.494 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:01:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 154.93698004000001, 2024-08-30T16:01:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:01:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:56.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:01:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:01:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:01:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:01:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:01:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:01:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:01:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:01:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:01:56.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:01:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:01:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:01:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:01:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:56.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:01:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:01:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:56.499 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:56.499 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:56.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:01:56.507 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:01:56.507 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:01:56.554 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:56.553+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:56.582 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:08.309 40108== Warning: unimplemented fcntl command: 1036 2024-08-30T16:01:56.600 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:01:56.600 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:01:56.638 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:56.638 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:56.638 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:56.638 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:56.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:01:56.655 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:56.653+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:56.688 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:01:56.689 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:01:56.756 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:56.754+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:56.772 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:01:56.772 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:01:56.814 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:56.814 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:56.814 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:56.815 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:56.815 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:56.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:01:56.852 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:01:56.853 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:01:56.857 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:56.856+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:01:56.960 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:56.956+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 151.585061349, 2024-08-30T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:56.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:01:57.059 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:57.057+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:57.155 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:57.155 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:57.155 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:01:57.155 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:57.159 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:57.158+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:57.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:01:57.260 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:57.259+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:57.290 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-08-30T16:01:57.346 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:57.346 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:01:57.346 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:01:57.347 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:01:57.347 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:57.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:01:57.398 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:57.395+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:57.462 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:57.461+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:57.563 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:01:57.563 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:01:57.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:01:57.563 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:01:57.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.563 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:57.564 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:57.564 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 153.11062154699999, 2024-08-30T16:01:57.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:01:57.564 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:57.564 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:57.564 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:57.564 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:57.564 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:57.564 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:57.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.564 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:57.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:57.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:57.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:57.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:01:57.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:57.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:01:57.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:01:57.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:57.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:01:57.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:01:57.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:57.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:01:57.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:01:57.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:01:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:01:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:01:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:01:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:01:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:01:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:01:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:57.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:01:57.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:01:57.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:57.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:01:57.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:01:57.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:01:57.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:01:57.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:01:57.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:57.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:01:57.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:01:57.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:57.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:01:57.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:01:57.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:57.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:01:57.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:01:57.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:57.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:01:57.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:01:57.571 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.571 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:57.571 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.571 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:01:57.572 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:57.572 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 151.994755212, 2024-08-30T16:01:57.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:01:57.572 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:57.572 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:57.572 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:57.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:57.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:57.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:57.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:01:57.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:57.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:01:57.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:57.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:57.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:01:57.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:01:57.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:57.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:01:57.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:01:57.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:57.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:01:57.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:01:57.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:57.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:01:57.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:01:57.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:57.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:01:57.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:01:57.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:57.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:01:57.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:01:57.576 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.576 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:57.576 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.576 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:01:57.576 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:57.576 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 151.935163714, 2024-08-30T16:01:57.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:01:57.576 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:57.577 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:57.577 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:57.577 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:57.577 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:57.577 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:01:57.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.577 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:57.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:57.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:57.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:57.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:01:57.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:57.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:01:57.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:01:57.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:57.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:01:57.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:01:57.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:57.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:01:57.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:01:57.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:57.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:01:57.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:01:57.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:57.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:01:57.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:01:57.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:57.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:01:57.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:01:57.580 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.581 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:57.581 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.581 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:01:57.581 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:57.581 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 151.90846502700001, 2024-08-30T16:01:57.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:01:57.582 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:57.582 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:57.582 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:57.582 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:57.582 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:57.582 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:57.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.582 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:57.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:57.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:57.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:57.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:01:57.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:57.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:01:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:01:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:01:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:01:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:01:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:57.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:01:57.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:01:57.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:57.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:01:57.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:01:57.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:57.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:01:57.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:01:57.585 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.586 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:57.586 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.586 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:57.586 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:57.586 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 151.04084849899999, 2024-08-30T16:01:57.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:01:57.586 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:57.586 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:57.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:57.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:57.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:57.587 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:57.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.587 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:57.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:57.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:57.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:57.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:01:57.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:57.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:01:57.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:57.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:57.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:01:57.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:01:57.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:57.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:01:57.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:01:57.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:57.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:01:57.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:01:57.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:57.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:01:57.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:01:57.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:57.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:01:57.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:01:57.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:57.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:01:57.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:01:57.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:01:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:01:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:01:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:01:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:01:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:01:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:57.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:01:57.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:01:57.592 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.592 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:57.592 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.592 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:01:57.592 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:57.592 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 150.88160142000001, 2024-08-30T16:01:57.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:01:57.593 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:57.593 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:57.593 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:57.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:57.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:57.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:57.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:01:57.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:57.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:01:57.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:01:57.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:57.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:01:57.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:01:57.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:57.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:01:57.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:01:57.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:57.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:01:57.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:01:57.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:57.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:01:57.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:01:57.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:57.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:01:57.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:01:57.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:57.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:01:57.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:01:57.596 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.596 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:57.596 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.597 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:57.597 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:57.597 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 150.605427354, 2024-08-30T16:01:57.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:01:57.597 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:57.597 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:57.597 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:57.597 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:57.597 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:57.597 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:01:57.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.597 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:57.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:57.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:57.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:57.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:01:57.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:57.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:01:57.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:01:57.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:57.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:01:57.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:01:57.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:57.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:01:57.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:01:57.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:57.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:01:57.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:01:57.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:57.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:01:57.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:01:57.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:57.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:01:57.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:01:57.600 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.600 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:57.601 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.601 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:57.601 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:57.601 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 149.94713577499999, 2024-08-30T16:01:57.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:01:57.601 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:57.601 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:57.601 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:57.601 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:01:57.601 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:01:57.601 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:01:57.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.602 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:57.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:57.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:57.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:57.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:01:57.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:57.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:01:57.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:57.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:57.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:01:57.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:01:57.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:57.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:01:57.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:01:57.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:57.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:01:57.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:01:57.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:57.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:01:57.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:01:57.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:57.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:01:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:01:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:01:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:01:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:01:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:01:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:01:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:01:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:01:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:01:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:01:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:01:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:01:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:01:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:01:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:01:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.606 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:57.607 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.607 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:01:57.607 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:57.607 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 149.85122249099999, 2024-08-30T16:01:57.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:01:57.607 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:57.607 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:01:57.607 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:57.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:57.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:57.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:57.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:01:57.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:57.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:01:57.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:57.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:57.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:01:57.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:01:57.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:57.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:01:57.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:01:57.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:57.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:01:57.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:01:57.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:57.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:01:57.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:01:57.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:01:57.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:01:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:01:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:01:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:01:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:01:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 90.59986198, 2024-08-30T16:01:57.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:01:57.612 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:57.612 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:57.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:57.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:57.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:57.613 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:01:57.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.613 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:57.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:57.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:57.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:57.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:01:57.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:57.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:01:57.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:01:57.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:57.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:01:57.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:01:57.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:57.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:01:57.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:01:57.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:57.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:01:57.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:01:57.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:57.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:01:57.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:01:57.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:57.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:01:57.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:01:57.617 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.617 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:57.618 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.618 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:01:57.618 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:57.618 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.590757292999999, 2024-08-30T16:01:57.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:01:57.618 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:01:57.618 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:01:57.618 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:01:57.618 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:01:57.618 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:57.618 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:01:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:01:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:01:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:01:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:01:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:01:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:01:57.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:01:57.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:01:57.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:01:57.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:01:57.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:01:57.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:01:57.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:01:57.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:01:57.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:01:57.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:01:57.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:01:57.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:01:57.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:01:57.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:01:57.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:01:57.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:01:57.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:01:57.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:01:57.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:01:57.622 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.622 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:57.622 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.622 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:01:57.622 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:01:57.622 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:01:57.622 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:57.562+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:57.623 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:01:57.669 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:57.667+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:57.749 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:57.749 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:57.749 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:57.757 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:01:57.765 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:57.763+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:57.866 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:57.864+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:57.927 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:57.927 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:57.927 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:57.927 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:57.928 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:57.936 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:01:57.967 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:57.965+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:58.061 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:58.061 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:58.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:58.061 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:01:58.061 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:58.068 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:58.066+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:58.070 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:01:58.168 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:58.167+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:58.249 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:58.249 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:58.249 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:58.249 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:58.258 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:01:58.269 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:58.268+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:58.370 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:58.369+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:58.388 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:58.388 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:58.388 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:58.388 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:58.388 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:58.398 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:01:58.483 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:58.481+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:58.572 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:58.570+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:01:58.612 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:58.612 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:58.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:58.613 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:58.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.613 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:58.613 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:58.613 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 167.74972527200001, 2024-08-30T16:01:58.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:01:58.613 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.613 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.613 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.613 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.613 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.614 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:01:58.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.614 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:58.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:01:58.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:01:58.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:01:58.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:01:58.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:01:58.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:01:58.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:01:58.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:01:58.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:01:58.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:01:58.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:01:58.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:01:58.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:01:58.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:01:58.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:01:58.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:01:58.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:01:58.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:01:58.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:01:58.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:01:58.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:01:58.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:01:58.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:01:58.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:01:58.619 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.619 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.619 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.619 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:58.619 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:58.619 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.83330925199999, 2024-08-30T16:01:58.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:01:58.619 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.619 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.620 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.620 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.620 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.620 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:01:58.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.620 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:58.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:01:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:01:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:01:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:01:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:01:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:01:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:01:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:01:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:01:58.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:01:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:01:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:01:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:01:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:01:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:01:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:01:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:01:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:01:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:01:58.624 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.831604699, 2024-08-30T16:01:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:01:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:01:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:01:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:01:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:01:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:01:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:01:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:01:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:01:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:01:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:01:58.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:01:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:01:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:01:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:01:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:01:58.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:01:58.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:01:58.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:01:58.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:01:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:01:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.412150892, 2024-08-30T16:01:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:01:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.630 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:01:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:01:58.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:01:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:01:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:01:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:01:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:01:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:01:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:01:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:01:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:01:58.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:01:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:01:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:01:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:01:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:01:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:01:58.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:01:58.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:01:58.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:01:58.635 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.635 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.635 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.635 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.635 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:58.636 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.395262718, 2024-08-30T16:01:58.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:01:58.636 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.636 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.636 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.636 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.636 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.636 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:01:58.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.636 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:58.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:01:58.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:01:58.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:01:58.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:01:58.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:01:58.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:01:58.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:01:58.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:01:58.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:01:58.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:01:58.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:01:58.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:01:58.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:01:58.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:01:58.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:01:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:01:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:01:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:01:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:01:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.641 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.641 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:58.641 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.38226264900001, 2024-08-30T16:01:58.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:01:58.641 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.641 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.641 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.641 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.641 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.641 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:01:58.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.641 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:58.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:01:58.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:01:58.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:01:58.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:01:58.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:01:58.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:01:58.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:01:58.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:01:58.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:01:58.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:01:58.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:01:58.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:01:58.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:01:58.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:01:58.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:01:58.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:01:58.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:01:58.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:01:58.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:01:58.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.646 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.646 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.646 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:58.646 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.37977035700001, 2024-08-30T16:01:58.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:01:58.646 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.646 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.646 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.647 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.647 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.647 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:01:58.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.647 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:58.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:01:58.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:01:58.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:01:58.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:01:58.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:01:58.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:01:58.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:01:58.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:01:58.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:01:58.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:01:58.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:01:58.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:01:58.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:01:58.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:01:58.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:01:58.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:01:58.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:01:58.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:01:58.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:01:58.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.653 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.654 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.654 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:58.654 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.37901290900001, 2024-08-30T16:01:58.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:01:58.654 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.654 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.654 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:01:58.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.654 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:58.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:01:58.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:01:58.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:01:58.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:01:58.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:01:58.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:01:58.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:01:58.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:01:58.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:01:58.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:01:58.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:01:58.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:01:58.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:01:58.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:01:58.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:01:58.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:01:58.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:01:58.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:01:58.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:01:58.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.659 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.659 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.659 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:58.659 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.378280459, 2024-08-30T16:01:58.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:01:58.660 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.660 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.660 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.660 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.660 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.660 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:01:58.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.660 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:58.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:01:58.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:01:58.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:01:58.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:01:58.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:01:58.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:01:58.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:01:58.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:01:58.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:01:58.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:01:58.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:01:58.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:01:58.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:01:58.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:01:58.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:01:58.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:01:58.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:01:58.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:01:58.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:01:58.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.664 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.664 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.665 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.665 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:58.665 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.37574165999999, 2024-08-30T16:01:58.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:01:58.665 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.665 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.665 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.665 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:01:58.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.665 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:58.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:01:58.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:01:58.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:58.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:01:58.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:01:58.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:01:58.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:01:58.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:01:58.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:01:58.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:01:58.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:01:58.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:01:58.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:01:58.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:01:58.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:01:58.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:01:58.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:01:58.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.670 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:01:58.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:01:58.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.670 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.670 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.670 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.670 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:58.670 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 166.37499365900001, 2024-08-30T16:01:58.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:01:58.670 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.670 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.671 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.671 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.671 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.671 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:01:58.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.671 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:58.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:01:58.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:01:58.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:01:58.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:01:58.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:01:58.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:01:58.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:01:58.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:01:58.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:01:58.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:01:58.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:01:58.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:01:58.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:01:58.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:01:58.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:01:58.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:01:58.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:01:58.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:01:58.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:01:58.675 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.675 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.675 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.675 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.675 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:58.676 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 165.60287718699999, 2024-08-30T16:01:58.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:01:58.676 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.676 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.676 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.676 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.676 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.676 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:01:58.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.676 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:58.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:01:58.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.677 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.677 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:01:58.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:01:58.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:01:58.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:01:58.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:01:58.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:01:58.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:01:58.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:01:58.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:01:58.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:01:58.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:01:58.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:01:58.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:01:58.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:01:58.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:01:58.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:01:58.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:01:58.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:01:58.681 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.681 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.681 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.681 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.681 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:58.681 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 165.60082004099999, 2024-08-30T16:01:58.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:01:58.681 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.681 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.682 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.682 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.682 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.682 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:01:58.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.682 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:58.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:01:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:01:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:01:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:01:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:01:58.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:01:58.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:01:58.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:01:58.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:01:58.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:01:58.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:01:58.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:01:58.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:01:58.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:01:58.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:01:58.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:01:58.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:01:58.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:01:58.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:01:58.686 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.686 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.686 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.686 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.687 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:58.687 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 165.33787554099999, 2024-08-30T16:01:58.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:01:58.687 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.687 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.687 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.687 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.687 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.687 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:01:58.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.687 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:58.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:01:58.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:01:58.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:01:58.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:01:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:01:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:01:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:01:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:01:58.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:01:58.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:01:58.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:01:58.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:01:58.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:01:58.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:01:58.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:01:58.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:01:58.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:01:58.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:01:58.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:01:58.692 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.692 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.692 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.692 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.692 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:58.692 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 165.33678403799999, 2024-08-30T16:01:58.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:01:58.693 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.693 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.693 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.693 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.693 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.693 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:01:58.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.693 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:58.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:01:58.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:01:58.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:01:58.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:01:58.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:01:58.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:01:58.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:01:58.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:01:58.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:01:58.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:01:58.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:01:58.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:01:58.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:01:58.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:01:58.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:01:58.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:01:58.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:01:58.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:01:58.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:01:58.697 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.697 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.698 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.698 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.698 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:58.698 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 165.336034453, 2024-08-30T16:01:58.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:01:58.698 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.698 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.698 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.698 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.698 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.698 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:01:58.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.699 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:58.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:01:58.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:01:58.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:58.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:01:58.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:01:58.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:01:58.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:01:58.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:01:58.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:01:58.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:01:58.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:01:58.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:01:58.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:01:58.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:01:58.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:01:58.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:01:58.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:01:58.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:01:58.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:01:58.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.703 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.703 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.703 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:58.703 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 165.30230292100001, 2024-08-30T16:01:58.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:01:58.704 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.704 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.704 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:01:58.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.704 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:58.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:01:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:01:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:01:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:01:58.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:01:58.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:01:58.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:01:58.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:01:58.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:01:58.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:01:58.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:01:58.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:01:58.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:01:58.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:01:58.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:01:58.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:01:58.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:01:58.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:01:58.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:01:58.708 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.708 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.708 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.709 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.709 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:58.709 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 165.301776913, 2024-08-30T16:01:58.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:01:58.709 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.709 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.709 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.709 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.709 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.710 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:01:58.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.710 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:58.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:01:58.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:01:58.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:58.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:01:58.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:01:58.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:01:58.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:01:58.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:01:58.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:01:58.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:01:58.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:01:58.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:01:58.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:01:58.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:01:58.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:01:58.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:01:58.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:01:58.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:01:58.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:01:58.714 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.714 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.714 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.714 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:01:58.714 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:58.714 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 165.30132134999999, 2024-08-30T16:01:58.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:01:58.714 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.715 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.715 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.715 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.715 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.715 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:01:58.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.715 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:58.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:01:58.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:01:58.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:01:58.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:01:58.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:01:58.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:01:58.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:01:58.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:01:58.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:01:58.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.717 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.717 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:01:58.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:01:58.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:01:58.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:01:58.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:01:58.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:01:58.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:01:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:01:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:01:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:01:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:01:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 160.63080603399999, 2024-08-30T16:01:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:01:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:01:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:01:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:01:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:01:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:01:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:01:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:01:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:01:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:01:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:01:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:01:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:01:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:01:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:01:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:01:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:01:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:01:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:01:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:01:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:01:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:01:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:01:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:01:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:58.725 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:58.726 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:01:58.727 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:01:58.725+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:01:58.828 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:58.826+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:58.891 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:58.891 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:58.891 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:58.901 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:01:58.942 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:58.940+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:58.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:01:59.029 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:59.028+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:59.071 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:59.072 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:59.072 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:59.072 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:59.072 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:59.081 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:01:59.130 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:01:59.129+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:01:59.210 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:01:59.211 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:01:59.239 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:01:59.237+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:01:59.251 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:59.251 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:59.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:59.252 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:59.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.252 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:01:59.252 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:59.252 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 152.72191455000001, 2024-08-30T16:01:59.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:01:59.252 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:59.252 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:59.252 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:59.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:59.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:59.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:59.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:59.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:01:59.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:59.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:59.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:01:59.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:01:59.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:59.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:01:59.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:01:59.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:59.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:01:59.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:01:59.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:59.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:01:59.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:01:59.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:59.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:01:59.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:01:59.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:59.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:01:59.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:01:59.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:59.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:01:59.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:01:59.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:59.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:01:59.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:01:59.256 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:59.256 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:59.256 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:59.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.256 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:01:59.257 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:59.257 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 151.60346640399999, 2024-08-30T16:01:59.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:01:59.257 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:59.257 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:01:59.257 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:59.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:59.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:59.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:59.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:59.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:01:59.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:59.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:59.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:01:59.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:01:59.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:59.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:01:59.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:01:59.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:59.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:01:59.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:01:59.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:59.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:01:59.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:01:59.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:59.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:01:59.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:01:59.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:01:59.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:01:59.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:01:59.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:01:59.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:01:59.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:01:59.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:59.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:01:59.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:01:59.261 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:59.261 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:59.261 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:59.261 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:59.261 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:59.261 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:59.262 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:01:59.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-08-30T16:01:59.309 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:01:59.310 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:01:59.332 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:59.331+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:59.388 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:59.389 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:01:59.389 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:59.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:01:59.398 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:01:59.401 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:01:59.401 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:01:59.440 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:59.438+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:59.512 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:01:59.512 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:01:59.537 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:01:59.536+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:01:59.565 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:01:59.565 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:01:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:01:59.566 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:59.574 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:01:59.600 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:01:59.600 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:01:59.658 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:01:59.656+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:01:59.686 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:01:59.687 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:01:59.736 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:59.734+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:59.744 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:01:59.745 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:01:59.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:01:59.745 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:01:59.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.745 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:01:59.745 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:59.745 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 151.364980968, 2024-08-30T16:01:59.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:01:59.745 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:59.746 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:59.746 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:59.746 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:59.746 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:59.746 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:01:59.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.746 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:59.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:59.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:59.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:59.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:59.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:01:59.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:59.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:59.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:01:59.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:01:59.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:59.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:01:59.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:01:59.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:59.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:01:59.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:01:59.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:59.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:01:59.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:01:59.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:59.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:01:59.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:01:59.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:59.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:01:59.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:01:59.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:59.750 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:59.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:59.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.751 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:01:59.751 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:59.751 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 31.251984852, 2024-08-30T16:01:59.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:01:59.751 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:01:59.751 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:01:59.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:01:59.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:01:59.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:01:59.751 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:01:59.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.751 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:01:59.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:01:59.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:59.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:59.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:01:59.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:01:59.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:01:59.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:01:59.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:01:59.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:01:59.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:01:59.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:01:59.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:01:59.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:01:59.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:01:59.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:01:59.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:01:59.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:01:59.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:01:59.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:01:59.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:01:59.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:01:59.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:01:59.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:01:59.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:01:59.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:01:59.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:01:59.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:59.755 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:59.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:59.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:01:59.755 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:01:59.755 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:01:59.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:01:59.771 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:01:59.771 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:01:59.836 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:01:59.834+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:01:59.853 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:01:59.853 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:01:59.918 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:01:59.918 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:01:59.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:01:59.936 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:01:59.935+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:00.037 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:00.036+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:00.084 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:00.084 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:00.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:00.140 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:00.139+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:00.213 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:02:00.226 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-08-30T16:02:00.226 INFO:tasks.mon_recovery:stopping all monitors 2024-08-30T16:02:00.227 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-30T16:02:00.227 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:02:00.239 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:00.239 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:00.239 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:00.238+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:00.241 INFO:tasks.ceph.mon.b.smithi094.stderr:2024-08-30T16:02:00.229+0000 a084640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 33042) UID: 0 2024-08-30T16:02:00.243 INFO:tasks.ceph.mon.b.smithi094.stderr:2024-08-30T16:02:00.230+0000 a084640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-08-30T16:02:00.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:00.340 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:00.339+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:00.427 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:00.427 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:00.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:00.479 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:00.476+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:00.542 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:00.541+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:00.605 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:00.605 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:00.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:00.643 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:00.641+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:00.742 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:00.742 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:00.744 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:00.742+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:00.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:00.845 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:00.843+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:00.929 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:00.929 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:00.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:00.944 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:00.943+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:01.046 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:01.044+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:01.106 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:01.106 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:01.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:01.134 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:02:01.134 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:02:01.147 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:01.145+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:01.242 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:01.243 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:01.247 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:01.246+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:01.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:01.348 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:01.347+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:01.380 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:01.380 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:01.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:01.449 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:01.448+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:01.550 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:01.549+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:01.557 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:01.557 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:01.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:01.651 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:01.650+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:01.694 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:01.694 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:01.703 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:01.752 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:01.751+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:01.840 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:01.840 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:01.849 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:01.852 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:01.851+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:01.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:02:01.960 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:01.958+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:01.973 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:01.973 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:01.981 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:02.066 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:02.064+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:02.147 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:02.147 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:02.155 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:02.154+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:02.157 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:02.256 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:02.255+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:02.320 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:02.320 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:02.331 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:02.373 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:02.371+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:02.459 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:02.457+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:02.495 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:02.495 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:02.505 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:02.589 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:02.587+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:02.661 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:02.659+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:02.669 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:02.669 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:02.678 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:02.761 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:02.760+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:02.842 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:02.842 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:02.851 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:02.897 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:02.895+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:02.963 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:02.962+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:02.976 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:02.976 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:02.985 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:03.071 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:03.069+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:03.150 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:03.150 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:03.160 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:03.202 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:03.200+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:03.265 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:03.264+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:03.324 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:03.324 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:03.333 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:03.378 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:03.376+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:03.458 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:03.458 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:03.467 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:03.467 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:03.465+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:03.567 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:03.566+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:03.636 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:03.636 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:03.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:03.687 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:03.685+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:03.772 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:03.768+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:03.789 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:03.789 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:03.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:03.882 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:03.881+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:03.972 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:03.970+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:03.974 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:03.974 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:03.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:04.072 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:04.071+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:04.151 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:04.151 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:04.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:04.201 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:04.200+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:04.274 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:04.273+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:04.329 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:04.329 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:04.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:04.379 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:04.378+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:04.479 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:04.478+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:04.505 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:04.505 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:04.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:04.581 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:04.580+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:04.648 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:04.649 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:04.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:04.700 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:04.698+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:04.783 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:04.782+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:04.828 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:04.829 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:04.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:04.884 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:04.883+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:04.968 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:04.968 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:04.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:04.985 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:04.983+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:05.086 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:05.085+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:05.145 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:05.145 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:05.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:05.187 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:05.185+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:05.288 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:05.286+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:05.325 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:05.325 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:05.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:05.388 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:05.387+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:05.489 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:05.487+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:05.502 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:05.502 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:05.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:05.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-08-30T16:02:05.589 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:05.588+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:05.680 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:05.680 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:05.690 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:05.690 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:05.688+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:05.791 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:05.790+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:05.826 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:05.826 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:05.835 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:05.892 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:05.890+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:05.960 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:05.960 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:05.969 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:06.013 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:06.011+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:06.094 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:06.092+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:06.134 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:02:06.135 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:02:06.135 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 7] 2024-08-30T16:02:06.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-08-30T16:02:06.136 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:06.136 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:06.146 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:06.194 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:06.193+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:06.295 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:06.294+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:06.313 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:06.313 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:06.322 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:06.329 INFO:tasks.ceph.mon.b:Stopped 2024-08-30T16:02:06.329 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-30T16:02:06.329 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:02:06.344 INFO:tasks.ceph.mon.c.smithi050.stderr:2024-08-30T16:02:06.332+0000 a084640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 38873) UID: 0 2024-08-30T16:02:06.345 INFO:tasks.ceph.mon.c.smithi050.stderr:2024-08-30T16:02:06.333+0000 a084640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-08-30T16:02:06.396 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:06.395+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:06.489 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:06.489 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:06.498 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:06.499 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:06.496+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:06.599 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:06.597+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:06.628 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:06.629 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:06.637 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:06.721 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:06.719+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:06.800 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:06.799+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:06.801 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:06.801 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:06.811 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:06.902 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:06.900+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:06.976 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:06.976 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:06.985 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:07.002 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:07.001+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:07.103 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:07.102+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:07.149 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:07.149 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:07.160 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:07.204 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:07.203+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:07.286 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:07.286 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:07.296 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:07.341 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:07.339+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:07.406 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:07.404+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:07.420 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:07.421 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:07.431 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:07.507 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:07.505+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:07.597 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:07.598 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:07.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:07.610 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:07.606+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:07.709 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:07.707+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:07.740 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:07.741 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:07.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:07.809 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:07.808+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:07.910 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:07.909+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:07.919 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:07.919 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:07.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:08.012 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:08.010+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:08.055 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:08.055 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:08.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:08.113 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:08.111+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:08.195 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:08.196 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:08.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:08.249 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:08.247+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:08.314 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:08.313+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:08.332 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:08.332 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:08.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:08.425 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:08.424+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:08.508 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:08.509 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:08.516 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:08.515+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:08.517 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:08.617 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:08.616+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:08.687 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:08.687 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:08.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:08.719 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:08.717+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:08.820 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:08.819+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:08.823 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:08.823 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:08.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:08.921 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:08.920+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:09.003 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:09.003 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:09.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:09.057 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:09.053+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:09.124 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:09.122+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:09.183 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:09.183 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:09.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:09.224 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:09.223+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:09.320 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:09.320 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:09.325 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:09.324+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:09.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:09.426 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:09.424+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:09.500 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:09.501 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:09.507 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:09.526 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:09.525+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:09.627 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:09.626+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:09.643 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:09.644 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:09.653 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:09.727 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:09.726+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:09.817 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:09.817 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:09.826 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:09.829 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:09.827+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:09.930 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:09.928+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:09.950 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:09.950 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:09.959 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:10.031 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:10.030+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:10.083 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:10.084 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:10.093 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:10.137 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:10.135+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:10.218 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:10.218 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:10.227 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:10.269 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:10.267+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:10.334 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:10.332+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:10.394 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:10.394 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:10.403 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:10.444 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:10.443+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:10.536 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:10.535+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:10.567 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:10.567 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:10.576 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:10.638 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:10.636+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:10.701 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:10.701 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:10.711 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:10.739 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:10.738+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:10.840 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:10.838+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:10.875 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:10.875 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:10.884 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:10.940 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:10.939+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:11.041 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:11.040+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:11.048 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:11.049 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:11.057 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:11.147 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:11.146+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:11.223 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:11.223 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:11.233 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:11.244 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:11.243+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:11.346 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:11.344+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:11.397 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:11.397 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:11.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:11.450 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:11.449+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:11.547 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:11.547 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:11.547 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:11.546+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:11.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:11.647 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-08-30T16:02:11.647 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-08-30T16:02:11.648 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:11.647+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:11.726 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:11.726 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:11.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:11.749 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:11.748+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:11.850 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:11.849+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:11.862 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:11.862 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:11.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:11.951 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:11.950+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:12.038 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:12.039 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:12.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:12.052 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:12.050+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:12.164 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:12.163+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:12.177 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:12.177 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:12.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:12.254 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:12.253+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:12.355 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:12.355 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:12.355 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:12.354+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:12.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:12.431 INFO:tasks.ceph.mon.c:Stopped 2024-08-30T16:02:12.431 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-30T16:02:12.432 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:02:12.456 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:12.455+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:12.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T16:02:12.450+0000 a084640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 40103) UID: 0 2024-08-30T16:02:12.463 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T16:02:12.452+0000 a084640 -1 mon.a@0(probing) e1 *** Got Signal Terminated *** 2024-08-30T16:02:12.535 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:12.535 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:12.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:12.558 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:12.556+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:12.658 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:12.657+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:12.712 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:12.713 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:12.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:12.764 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:12.763+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:12.860 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:12.858+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:12.891 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:12.891 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:12.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:12.961 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:12.959+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:13.030 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:13.030 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:13.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:13.085 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:13.083+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:13.163 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:13.161+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:13.167 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:13.167 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:13.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:13.263 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:13.262+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:13.346 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:13.346 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:13.355 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:13.363 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:13.362+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:13.469 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:13.468+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:13.496 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:13.496 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:13.505 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:13.570 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:13.569+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:13.635 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:13.635 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:13.645 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:13.671 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:13.670+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:13.770 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:13.770 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:13.771 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:13.770+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:13.779 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:13.872 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:13.871+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:13.944 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:13.944 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:13.953 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:13.994 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:13.992+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:14.075 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:14.072+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:14.116 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:14.116 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:14.125 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:14.210 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:14.208+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:14.276 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:14.274+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:14.288 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:14.288 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:14.297 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:14.377 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:14.375+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:14.461 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:14.461 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:14.470 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:14.477 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:14.476+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:14.578 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:14.577+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:14.595 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:14.595 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:14.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:14.679 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:14.678+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:14.768 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:14.768 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:14.777 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:14.779 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:14.777+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:14.881 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:14.879+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:14.900 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:14.900 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:14.909 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:14.982 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:14.980+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:15.034 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:15.034 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:15.047 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:15.092 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:15.090+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:15.172 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:15.172 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:15.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:15.183 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:15.182+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:15.284 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:15.283+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:15.322 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:15.322 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:15.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:15.385 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:15.383+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:15.486 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:15.484+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:15.500 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:15.500 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:15.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:15.587 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:15.585+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:15.636 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:15.636 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:15.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:15.687 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:15.686+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:15.788 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:15.786+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:15.813 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:15.813 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:15.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:15.889 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:15.887+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:15.950 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:15.950 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:15.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:15.990 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:15.988+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:16.086 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:16.087 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:16.090 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:16.089+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:16.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:16.191 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:16.189+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:16.224 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:16.224 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:16.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:16.318 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:16.317+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:16.392 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:16.391+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:16.403 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:16.403 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:16.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:16.496 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:16.495+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:16.579 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:16.579 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:16.587 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:16.630 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:16.627+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:16.695 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:16.693+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:16.755 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:16.755 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:16.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:16.806 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:16.805+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:16.896 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:16.895+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:16.931 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:16.932 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:16.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:16.998 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:16.996+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:17.098 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:17.097+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:17.115 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:17.115 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:17.125 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:17.199 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:17.198+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:17.275 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:17.275 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:17.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:17.302 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:17.300+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:17.402 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:17.401+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:17.465 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:17.465 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:17.475 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:17.503 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:17.501+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:17.598 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:17.599 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:17.603 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:17.601+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:17.607 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:17.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-08-30T16:02:17.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-08-30T16:02:17.704 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:17.701+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:17.731 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:17.731 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:17.740 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:17.805 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:17.803+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:17.864 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:17.865 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:17.874 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:17.906 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:17.903+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:17.997 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:17.998 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:18.006 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:18.007 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:18.004+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:18.107 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:18.105+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:18.130 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:18.130 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:18.139 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:18.225 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:18.223+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:18.305 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:18.305 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:18.309 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:18.307+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:18.315 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:18.410 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:18.408+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:18.485 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:18.485 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:18.494 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:18.534 INFO:tasks.ceph.mon.a:Stopped 2024-08-30T16:02:18.534 INFO:tasks.mon_recovery:forming a minimal quorum for ['b', 'c', 'a'], then adding monitors 2024-08-30T16:02:18.534 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-30T16:02:18.535 DEBUG:teuthology.orchestra.run.smithi094:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-08-30T16:02:18.536 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:18.533+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:18.537 INFO:tasks.ceph.mon.b:Started 2024-08-30T16:02:18.537 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-30T16:02:18.538 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-08-30T16:02:18.540 INFO:tasks.ceph.mon.c:Started 2024-08-30T16:02:18.540 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2024-08-30T16:02:18.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:02:18.611 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:18.609+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:18.661 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:18.661 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:18.670 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:18.715 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:18.713+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:18.796 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:18.796 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:18.805 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:18.812 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:18.810+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:18.913 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:18.910+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:18.970 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:18.970 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:18.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:19.024 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:19.021+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:19.107 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:19.107 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:19.115 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:19.113+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:19.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:19.224 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:19.219+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:19.284 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:19.284 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:19.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:19.316 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:19.314+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:19.417 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:19.415+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:19.435 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:19.436 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:19.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:19.518 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:19.516+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:19.617 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:19.617 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:19.618 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:19.616+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:19.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:19.720 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:19.717+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:19.796 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:19.797 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:19.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:19.821 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:19.819+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:19.922 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:19.921+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:19.975 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:19.975 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:19.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:20.024 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:20.022+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:20.113 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:20.113 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:20.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:20.124 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:20.122+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:20.225 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:20.223+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:20.255 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:20.255 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:20.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:20.326 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:20.325+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:20.394 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:20.394 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:20.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:20.447 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:20.445+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:20.528 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:20.526+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:20.531 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:20.531 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:20.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:20.630 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:20.628+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:20.707 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:20.708 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:20.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:20.757 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:20.755+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:20.831 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:20.829+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:20.883 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:20.884 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:20.893 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:20.939 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:20.937+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:21.021 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:21.021 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:21.031 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:21.033 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:21.030+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:21.133 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:21.131+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:21.201 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:21.201 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:21.210 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:21.255 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:21.253+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:21.335 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:21.333+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:21.336 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:21.336 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:21.345 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:21.436 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:21.435+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:21.511 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:21.511 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:21.521 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:21.561 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:21.560+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:21.638 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:21.636+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:21.687 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:21.687 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:21.696 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:21.740 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:21.737+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:21.840 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:21.837+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:21.861 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:21.862 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:21.871 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:21.940 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:21.938+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:21.996 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:21.996 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:22.005 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:22.041 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:22.039+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:22.129 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:22.129 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:22.138 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:22.184 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:22.182+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:22.244 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:22.242+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:22.267 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:22.267 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:22.276 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:22.361 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:22.359+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:22.445 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:22.445 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:22.446 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:22.443+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:22.454 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:22.547 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:22.544+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:22.621 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:22.621 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:22.630 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:22.675 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:22.673+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:22.749 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:22.747+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:22.758 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:22.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:22.849 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:22.847+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:22.898 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:22.898 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:22.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:22.950 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:22.948+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:23.051 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:23.050+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:23.075 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:23.075 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:23.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:23.152 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:23.150+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:23.216 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:23.216 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:23.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:23.267 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:23.265+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:23.354 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:23.352+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:23.395 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:23.395 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:23.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:23.455 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:23.453+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:23.549 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:23.549 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:23.555 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:23.554+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:23.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:23.657 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:23.655+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:23.689 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:23.689 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:23.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:23.757 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:23.756+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:23.825 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:23.825 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:23.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:23.858 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:23.856+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:23.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-08-30T16:02:23.959 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:23.958+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:23.962 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:23.962 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:23.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:24.060 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:24.057+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:24.101 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:24.101 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:24.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:24.161 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:24.159+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:24.262 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:24.260+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:24.278 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:24.278 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:24.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:24.362 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:24.360+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:24.415 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:24.415 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:24.424 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:24.463 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:24.462+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:24.580 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:24.577+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:24.593 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:24.593 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:24.603 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:24.666 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:24.664+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:24.729 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:24.730 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:24.739 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:24.766 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:24.764+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:24.868 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:24.866+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:24.905 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:24.905 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:24.914 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:24.968 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:24.966+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:25.069 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:25.067+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:25.080 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:25.080 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:25.090 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:25.170 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:25.168+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:25.219 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:25.219 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:25.228 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:25.272 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:25.269+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:25.353 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:25.353 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:25.362 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:25.407 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:25.405+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:25.473 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:25.471+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:25.503 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:25.503 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:25.513 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:25.575 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:25.572+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:25.676 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:25.673+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:25.681 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:25.681 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:25.690 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:25.777 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:25.775+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:25.829 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:25.829 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:25.838 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:25.881 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:25.878+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:25.978 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:25.977+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:26.005 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:26.005 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:26.015 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:26.040 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:07.445 36144== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:26.084 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:26.082+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:26.165 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:26.165 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:26.174 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:26.216 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:26.214+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:26.257 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:07.662 36144== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:26.281 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:26.279+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:26.338 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:26.339 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:26.348 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:26.389 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:26.387+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:26.390 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:07.780 42904== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:26.484 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:26.482+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:26.512 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:26.512 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:26.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:26.585 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:26.583+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:26.607 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:07.997 42904== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:26.650 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:26.650 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:26.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:26.686 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:26.684+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:26.786 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:26.787 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:26.787 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:26.784+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:26.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:26.888 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:26.886+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:26.964 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:26.964 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:26.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:27.017 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:27.014+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:27.090 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:27.087+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:27.101 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:27.101 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:27.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:27.194 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:27.192+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:27.277 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:27.277 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:27.287 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:27.328 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:27.326+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:27.392 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:27.390+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:27.458 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:27.458 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:27.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:27.493 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:27.491+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:27.594 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:27.593+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:27.595 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:27.595 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:27.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:27.695 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:27.694+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:27.735 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:27.735 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:27.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:27.796 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:27.794+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:27.871 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:27.871 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:27.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:27.896 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:27.895+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:27.997 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:27.996+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:28.008 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:28.008 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:28.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:28.098 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:28.097+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:28.166 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:28.166 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:28.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:28.199 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:28.198+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:28.300 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:28.299+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:28.304 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:28.304 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:28.313 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:28.401 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:28.399+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:28.453 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:28.453 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:28.463 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:28.502 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:28.501+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:28.603 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:28.602+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:28.628 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:28.628 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:28.637 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:28.704 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:28.702+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:28.764 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:28.765 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:28.772 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:28.805 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:28.804+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:28.907 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:28.905+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:28.935 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:28.935 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:28.944 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:29.008 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:29.006+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:29.068 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:29.068 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:29.077 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:29.109 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:29.107+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:29.210 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:29.208+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:29.253 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:29.253 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:29.263 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:29.310 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:29.308+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:29.390 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:29.390 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:29.399 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:29.411 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:29.410+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:29.513 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:29.511+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:29.566 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:29.566 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:29.575 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:29.614 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:29.612+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:29.700 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:29.700 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:29.709 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:29.753 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:29.751+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:29.815 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:29.813+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:29.834 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:29.834 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:29.843 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:29.916 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:29.914+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:30.008 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:30.008 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:30.016 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:30.015+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:30.017 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:30.117 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:30.115+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:30.141 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:30.141 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:30.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:30.217 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:30.216+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:30.282 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:30.283 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:30.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:30.336 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:30.335+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:30.419 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:30.418+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:30.421 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:30.421 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:30.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:30.520 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:30.519+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:30.598 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:30.598 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:30.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:30.652 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:30.650+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:30.722 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:30.721+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:30.735 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:30.736 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:30.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:30.829 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:30.828+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:30.913 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:30.913 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:30.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:30.924 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:30.923+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:31.026 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:31.024+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:31.092 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:31.092 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:31.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:31.127 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:31.125+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:31.228 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:31.227+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:31.231 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:31.232 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:31.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:31.329 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:31.328+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:31.408 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:31.408 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:31.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:31.459 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:31.458+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:31.531 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:31.529+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:31.589 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:31.589 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:31.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:31.632 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:31.631+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:31.733 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:31.732+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:31.768 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:31.768 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:31.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:31.833 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:31.832+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:31.905 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:31.905 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:31.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:31.957 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:31.954+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:32.036 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:32.035+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:32.082 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:32.083 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:32.092 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:32.138 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:32.136+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:32.235 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:32.235 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:32.238 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:32.237+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:32.245 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:32.338 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:32.337+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:32.410 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:32.410 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:32.419 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:32.464 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:32.462+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:32.540 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:32.539+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:32.548 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:32.548 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:32.558 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:32.642 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:32.640+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:32.724 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:32.725 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:32.734 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:32.779 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:32.777+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:32.843 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:32.841+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:32.862 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:32.862 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:32.872 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:32.957 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:32.955+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:33.036 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:33.036 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:33.044 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:33.043+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:33.046 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:33.145 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:33.144+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:33.210 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:33.211 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:33.220 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:33.246 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:33.245+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:33.343 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:33.343 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:33.347 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:33.346+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:33.352 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:33.448 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:33.447+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:33.517 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:33.517 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:33.526 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:33.550 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:33.548+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:33.650 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:33.649+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:33.691 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:33.786 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:33.787 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:33.788 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:33.786+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:33.888 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:33.887+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:33.912 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:33.912 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:33.921 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:33.989 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:33.988+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:34.045 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:34.045 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:34.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:34.099 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:34.097+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:34.191 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:34.189+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:34.194 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:34.194 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:34.204 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:34.291 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:34.290+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:34.372 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:34.372 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:34.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:34.392 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:34.390+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:34.493 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:34.492+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:34.563 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:34.563 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:34.573 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:34.595 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:34.593+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:34.695 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:34.694+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:34.742 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:34.742 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:34.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:34.797 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:34.795+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:34.884 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:34.885 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:34.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:34.937 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:34.934+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:34.998 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:34.997+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:35.066 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:35.066 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:35.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:35.101 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:35.099+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:35.202 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:35.201+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:35.251 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:35.252 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:35.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:35.305 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:35.304+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:35.396 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:35.396 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:35.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:35.406 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:35.403+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:35.505 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:35.504+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:35.586 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:35.586 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:35.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:35.606 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:35.605+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:35.708 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:35.706+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:35.763 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:35.763 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:35.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:35.809 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:35.807+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:35.910 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:35.909+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:35.942 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:35.942 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:35.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:36.011 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:36.010+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:36.080 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:36.080 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:36.090 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:36.136 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:36.134+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:36.213 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:36.211+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:36.228 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:36.228 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:36.237 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:36.322 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:36.320+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:36.331 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:02:36.331 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:02:36.407 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:36.407 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:36.414 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:36.413+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:36.415 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:36.430 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:02:36.431 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:02:36.516 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:36.514+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:36.520 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:02:36.521 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:02:36.583 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:36.583 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:36.592 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:36.611 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:02:36.611 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:02:36.640 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:36.638+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:36.699 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:02:36.699 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:02:36.718 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:36.717+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:36.819 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:36.818+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:36.824 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:02:36.824 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:02:36.825 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:36.826 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:36.826 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:02:36.826 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:36.835 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:02:36.921 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:36.919+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:37.007 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:02:37.007 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:02:37.010 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:37.010 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:37.010 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:02:37.010 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:02:37.010 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:37.021 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:37.026 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:37.023+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:37.121 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:37.120+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:37.168 INFO:tasks.ceph.mon.b.smithi094.stderr:2024-08-30T16:02:37.149+0000 1008f640 -1 mon.b@1(leader).osd e59 definitely_dead 0 2024-08-30T16:02:37.200 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":38,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:02:37.213 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2024-08-30T16:02:37.213 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-30T16:02:37.214 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-08-30T16:02:37.218 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:37.218 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:02:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:02:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:02:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:02:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:02:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 206.35573097599999, 2024-08-30T16:02:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:02:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.219 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:02:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:02:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:02:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:02:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:02:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:02:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:02:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:02:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:02:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:02:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:02:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:02:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:02:37.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:02:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:02:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:02:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:02:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:02:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:02:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:02:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:02:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:02:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:02:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:02:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:02:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:02:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:02:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:02:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 205.439314956, 2024-08-30T16:02:37.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:02:37.226 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.226 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.226 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.226 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.226 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.226 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:02:37.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.226 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:02:37.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:02:37.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:02:37.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:02:37.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:02:37.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:02:37.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:02:37.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:02:37.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:02:37.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:02:37.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:02:37.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:02:37.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:02:37.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:02:37.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:02:37.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:02:37.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:02:37.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:02:37.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:02:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:02:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:02:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:02:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 205.43761040300001, 2024-08-30T16:02:37.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:02:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:02:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:02:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:02:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:02:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:02:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:02:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:02:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:02:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:02:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:02:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:02:37.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:02:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:02:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:02:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:02:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:02:37.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:02:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:02:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:02:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:02:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:02:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.236 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:02:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 205.01815659600001, 2024-08-30T16:02:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:02:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.237 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:02:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:02:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:02:37.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:02:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:02:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:02:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:02:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:02:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:02:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:02:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:02:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:02:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:02:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:02:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:02:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:02:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:02:37.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:02:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:02:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:02:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:02:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:02:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 205.00126842200001, 2024-08-30T16:02:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:02:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:02:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.243 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:02:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:02:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:02:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:02:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:02:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:02:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:02:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:02:37.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:02:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:02:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:02:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:02:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:02:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:02:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:02:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:02:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:02:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:02:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:02:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:02:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:02:37.248 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.988268353, 2024-08-30T16:02:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:02:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:02:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.249 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:02:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:02:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:02:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:02:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:02:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:02:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:02:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:02:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:02:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:02:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:02:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:02:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:02:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:02:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:02:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:02:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:02:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:02:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:02:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:02:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:02:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.985776061, 2024-08-30T16:02:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:02:37.254 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:02:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:02:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:02:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:02:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:02:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:02:37.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:02:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:02:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:02:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:02:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:02:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:02:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:02:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:02:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:02:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:02:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:02:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:02:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:02:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:02:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:02:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:02:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.98501861299999, 2024-08-30T16:02:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:02:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:02:37.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:02:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:02:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:02:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:02:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:02:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:02:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:02:37.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:02:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:02:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:02:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:02:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:02:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:02:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:02:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:02:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:02:37.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:02:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:02:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:02:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:02:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.265 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:02:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.98428616300001, 2024-08-30T16:02:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:02:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.266 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:02:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:02:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:02:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:02:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:02:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:02:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:02:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:02:37.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:02:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:02:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:02:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:02:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:02:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:02:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:02:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:02:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:02:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:02:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:02:37.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:02:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:02:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:02:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.981747364, 2024-08-30T16:02:37.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:02:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:02:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:02:37.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:02:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:02:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:02:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:02:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:02:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:02:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:02:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:02:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:02:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:02:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:02:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:02:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:02:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:02:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:02:37.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:02:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:02:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:02:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:02:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:02:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.980999363, 2024-08-30T16:02:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:02:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:02:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.277 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:02:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:02:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:02:37.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:02:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:02:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:02:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:02:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:02:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:02:37.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:02:37.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:02:37.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:02:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:02:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:02:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:02:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:02:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:02:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:02:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:02:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:02:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.282 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:02:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.208882891, 2024-08-30T16:02:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:02:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:02:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:02:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:02:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:02:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:02:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:02:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:02:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:02:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:02:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:02:37.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:02:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:02:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:02:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:02:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:02:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:02:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:02:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:02:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:02:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:02:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:02:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.287 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:02:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 204.206825745, 2024-08-30T16:02:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:02:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.288 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:02:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:02:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:02:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:02:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:02:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:02:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:02:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:02:37.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:02:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:02:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:02:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:02:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:02:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:02:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:02:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:02:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:02:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:02:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:02:37.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:02:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:02:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:02:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 203.943881245, 2024-08-30T16:02:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:02:37.293 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:02:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:02:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:02:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:02:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:02:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:02:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:02:37.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:02:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:02:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:02:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:02:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:02:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:02:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:02:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:02:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:02:37.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:02:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:02:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:02:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:02:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:02:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.298 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:02:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 203.942789742, 2024-08-30T16:02:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:02:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:02:37.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:02:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:02:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:02:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:02:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:02:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:02:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:02:37.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:02:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:02:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:02:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:02:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:02:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.302 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:02:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:02:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:02:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:02:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:02:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:02:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:02:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:02:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:02:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 203.94204015700001, 2024-08-30T16:02:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:02:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:02:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:02:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:02:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:02:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:02:37.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:02:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:02:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:02:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:02:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:02:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:02:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:02:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:02:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:02:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:02:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:02:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:02:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:02:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:02:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.309 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:02:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:02:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:02:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 203.90830862499999, 2024-08-30T16:02:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:02:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.310 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:02:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:02:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.311 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:02:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:02:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:02:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:02:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:02:37.312 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:02:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:02:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:02:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:02:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.313 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:02:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:02:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:02:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:02:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:02:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:02:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.314 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:02:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:02:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:02:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:02:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:02:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 203.90778261700001, 2024-08-30T16:02:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:02:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.316 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:02:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:02:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:02:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:02:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:02:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:02:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:02:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:02:37.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:02:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:02:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:02:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:02:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:02:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:02:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:02:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:02:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:02:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:02:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:02:37.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:02:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:02:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:02:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 203.90732705400001, 2024-08-30T16:02:37.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:02:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:02:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:02:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:02:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:02:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:02:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:02:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:02:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:02:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:02:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:02:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:02:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:02:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:02:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:02:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:02:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:02:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:02:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:02:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:02:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:02:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:02:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.326 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:02:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:02:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.236811738, 2024-08-30T16:02:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:02:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:02:37.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:02:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:02:37.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:02:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:02:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:02:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:02:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:02:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:02:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:02:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:02:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:02:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:02:37.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:02:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:02:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:02:37.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:02:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:02:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:02:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:02:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:02:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.332 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.333 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.333 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:37.335 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-08-30T16:02:37.336 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:37.336 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:37.333+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:37.337 INFO:tasks.ceph.mon.a:Started 2024-08-30T16:02:37.337 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-08-30T16:02:37.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:02:37.435 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:37.433+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:37.462 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:37.462 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:37.462 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:02:37.462 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:37.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:37.536 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:37.535+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:37.637 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:37.636+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:37.640 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:02:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:02:37.640 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:37.649 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:37.738 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:37.737+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:37.782 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:37.782 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:02:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:02:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:02:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:02:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:02:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 191.25294360800001, 2024-08-30T16:02:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:02:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:02:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:02:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:02:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:02:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:02:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:02:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:02:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:02:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:02:37.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:02:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:02:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:02:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:02:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:02:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:02:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:02:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:02:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:02:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:02:37.787 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:02:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 190.13449546199999, 2024-08-30T16:02:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:02:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:02:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:37.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:02:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:02:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:02:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:02:37.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:02:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:37.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:02:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:02:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:02:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:02:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:02:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:02:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:02:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:02:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:02:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:02:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:37.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:02:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:02:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:37.793 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:37.793 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:37.794 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:37.839 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:37.838+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:37.943 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:37.942+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:37.966 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:37.966 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:02:37.966 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:37.976 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:38.041 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:38.040+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:38.054 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":38,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:02:38.110 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:38.111 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:38.111 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:02:38.111 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:02:38.111 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:38.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:38.125 INFO:teuthology.orchestra.run.smithi050.stderr:marked out osd.4. 2024-08-30T16:02:38.142 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:38.141+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:38.243 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:38.242+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:38.250 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:38.250 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:02:38.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:02:38.250 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:02:38.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.250 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:02:38.250 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:02:38.250 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.87013503, 2024-08-30T16:02:38.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:02:38.250 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:02:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:02:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:02:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:02:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:02:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:38.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:02:38.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:38.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:38.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:02:38.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:02:38.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:38.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:02:38.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:02:38.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:38.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:02:38.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:02:38.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:38.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:02:38.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:02:38.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:38.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:02:38.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:02:38.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:38.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:02:38.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:02:38.254 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:38.254 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:38.254 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:38.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.254 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:02:38.254 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:02:38.254 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.757138913999995, 2024-08-30T16:02:38.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:02:38.255 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:38.255 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:02:38.255 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:38.255 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:02:38.255 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:02:38.255 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:02:38.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.255 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:38.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:38.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:02:38.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:38.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:38.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:02:38.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:38.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:38.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:02:38.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:02:38.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:38.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:02:38.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:02:38.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:38.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:02:38.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:02:38.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:38.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:02:38.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:02:38.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:38.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:02:38.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:02:38.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:38.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:38.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:38.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:02:38.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:02:38.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:38.258 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:38.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:38.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:38.258 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:38.259 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:38.260 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:38.344 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:38.342+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:38.389 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:38.389 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:38.389 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:38.389 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:38.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:38.445 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:38.444+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:38.526 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:38.527 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:38.527 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:38.527 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:38.527 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:38.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:38.581 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:38.579+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:38.646 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:38.645+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:38.665 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:02:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:02:38.665 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:02:38.665 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:38.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:38.747 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:38.746+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:38.844 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:38.844 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:38.844 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:38.845 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:38.848 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:38.847+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:38.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:38.949 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:38.948+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:38.984 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:38.984 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:38.985 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:38.985 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:38.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:39.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:02:39.050 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:39.049+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:39.128 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:02:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:02:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:02:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:02:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 197.853726456, 2024-08-30T16:02:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:02:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:02:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:02:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:02:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:02:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:02:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:02:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:02:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:02:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:02:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:02:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:02:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:02:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:02:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 197.580799095, 2024-08-30T16:02:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:02:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:02:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:02:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:02:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:02:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:02:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:02:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:02:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:02:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:02:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:02:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:02:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:02:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:02:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:02:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:39.138 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:39.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:39.182 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:39.181+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:39.264 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:39.262+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:39.278 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:39.279 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:39.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:39.317 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:02:39.318 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:02:39.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-08-30T16:02:39.353 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:39.352+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:39.436 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:02:39.437 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:02:39.453 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:39.453+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:39.477 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:39.477 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:39.478 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:39.478 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:39.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:39.541 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:02:39.541 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:02:39.554 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:39.553+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:39.632 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:39.632 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:39.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:39.648 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:02:39.648 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:02:39.677 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:39.675+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:39.755 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:39.754+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:39.799 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:39.799 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:39.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:39.863 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:02:39.863 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:02:39.953 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:02:39.953 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:02:39.954 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:39.951+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:39.957 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:39.955+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:40.006 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:40.006 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:40.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:40.038 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:02:40.038 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:02:40.058 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:40.056+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:40.137 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:02:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:02:40.156 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:40.156 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:40.158 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:40.157+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:40.165 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:40.259 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:40.258+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:40.307 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-08-30T16:02:40.311 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:40.311 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:40.321 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:40.363 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:40.362+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:40.445 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:40.445 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:40.454 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:40.462 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:40.461+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:40.562 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:40.560+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:40.663 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:40.662+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:40.714 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:40.714 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:40.723 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:40.766 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:40.764+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:40.865 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:40.863+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:40.886 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:40.887 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:40.895 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:40.966 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:40.965+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:41.021 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:41.021 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:41.030 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:41.066 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:41.065+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:41.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:02:41.180 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:41.179+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:41.194 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:41.194 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:41.203 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:41.288 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:41.287+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:41.367 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:41.367 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:41.369 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:41.368+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:41.376 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:41.469 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:41.468+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:41.550 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:41.551 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:41.560 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:41.570 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:41.569+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:41.671 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:41.670+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:41.690 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":38,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:02:41.724 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:41.725 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:41.734 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:41.772 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:41.771+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:41.856 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:41.856 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:41.866 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:41.910 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:41.909+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:41.973 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:41.973+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:41.998 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:42.007 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:42.092 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:42.091+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:42.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:02:42.171 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:42.171 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:42.175 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:42.175+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:42.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:42.277 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:42.275+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:42.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-08-30T16:02:42.342 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:42.342 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:42.352 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:42.378 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:42.377+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:42.409 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:02:42.409 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:02:42.479 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:42.478+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:42.508 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:02:42.509 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:02:42.543 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:42.543 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:42.543 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:42.543 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:42.544 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:42.555 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:42.580 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:42.579+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:42.624 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:02:42.624 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:02:42.681 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:42.680+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:42.747 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:02:42.747 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:02:42.753 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:42.753 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:02:42.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:02:42.754 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:02:42.754 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:42.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:42.783 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:42.782+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:42.850 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:02:42.850 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:02:42.883 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:42.882+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:42.943 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:42.944 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:42.951 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:02:42.952 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:02:42.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:42.994 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:42.993+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:43.032 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:02:43.032 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:02:43.085 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:43.084+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:43.118 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:02:43.118 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:02:43.123 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:43.123 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:43.123 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:43.123 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:43.123 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:43.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:43.143 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7] out_osds: [6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:02:43.143 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:02:43.167 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:24.573 36144== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:43.217 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:43.215+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:43.267 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:24.673 36144== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:43.286 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:43.285+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:43.306 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:43.306 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:02:43.306 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:02:43.306 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:02:43.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.307 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:02:43.307 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:43.307 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 202.03174659300001, 2024-08-30T16:02:43.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:02:43.307 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:43.307 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:43.307 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:43.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.307 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:43.307 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:43.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:43.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.308 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:43.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:43.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:43.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.308 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:43.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:02:43.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:02:43.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.308 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:43.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:02:43.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:02:43.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.309 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:43.309 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:02:43.309 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:02:43.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.309 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:43.309 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:02:43.309 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:02:43.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:02:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:02:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:02:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:02:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:02:43.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:02:43.311 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:43.311 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:43.311 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:43.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.311 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:02:43.311 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:43.311 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 201.75881923200001, 2024-08-30T16:02:43.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:02:43.311 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:43.311 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:43.311 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:43.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:43.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:43.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:43.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:43.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:43.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:43.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:43.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:02:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:02:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:02:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:02:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:02:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:02:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.313 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:43.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:02:43.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:02:43.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.314 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:43.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:02:43.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:02:43.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.314 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:43.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:02:43.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:02:43.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:43.316 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:02:43.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:02:43.316 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:43.316 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:43.316 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:43.316 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:43.316 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:43.316 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:43.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:43.347 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:24.737 42904== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:43.392 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:43.391+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:43.405 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:24.795 42904== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:43.454 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:24.859 36144== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:43.478 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:43.478 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:43.478 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:43.478 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:43.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:43.488 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:43.486+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:43.589 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:43.588+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:43.628 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:25.018 42904== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:43.656 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:43.656 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:43.656 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:43.656 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:43.656 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:43.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:43.709 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:43.707+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:43.771 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:25.177 36144== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:43.772 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:25.178 36144== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:43.782 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:25.188 36144== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:43.790 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:43.789+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:43.795 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:43.795 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:02:43.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:02:43.795 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:02:43.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.795 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:02:43.795 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:02:43.795 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 198.41757799300001, 2024-08-30T16:02:43.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:02:43.795 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:43.796 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:43.796 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:43.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.796 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:43.796 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:02:43.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:43.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.796 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:43.796 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:02:43.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:43.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:02:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:02:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:02:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:02:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:02:43.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:02:43.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:43.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:02:43.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:02:43.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:43.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:02:43.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:02:43.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:02:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:02:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:02:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:02:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:02:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:02:43.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:02:43.800 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:43.800 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:43.800 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:43.800 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:43.800 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:43.800 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:43.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:43.891 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:43.891+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:43.977 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:43.978 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:43.978 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:43.978 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:43.985 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:25.375 42904== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:43.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:43.987 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:25.377 42904== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:43.992 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:43.991+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:44.058 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:25.448 42904== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:44.093 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:44.093+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:44.119 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-08-30T16:02:44.128 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:44.128 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:44.128 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:44.128 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:44.128 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:44.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:44.225 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:44.223+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:44.295 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:44.295+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:44.340 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:02:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:02:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:02:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:02:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:02:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 199.885149117, 2024-08-30T16:02:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:02:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:02:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:02:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:02:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:02:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:02:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:02:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:02:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:02:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:02:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:02:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:02:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:02:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:02:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:02:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:02:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:02:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:02:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:02:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:02:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:02:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:02:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:02:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:02:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:02:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:02:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:02:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:02:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:02:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:02:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:02:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:02:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:02:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:02:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:02:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:02:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:02:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:02:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:02:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 198.769282782, 2024-08-30T16:02:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:02:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:02:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:02:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:02:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:02:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:02:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:02:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:02:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:02:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:02:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:02:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:02:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:02:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:02:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:02:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:02:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:02:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:02:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:02:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 198.709691284, 2024-08-30T16:02:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:02:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:02:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:02:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:02:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:02:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:02:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:02:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:02:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:02:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:02:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:02:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:02:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:02:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:02:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:02:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:02:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:02:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:02:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:02:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:02:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:02:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 198.68299259700001, 2024-08-30T16:02:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:02:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:02:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:02:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:02:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:02:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:02:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:02:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:02:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:02:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:02:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:02:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:02:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:02:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:02:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:02:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:02:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:02:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:02:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:02:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:02:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:02:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 197.815376069, 2024-08-30T16:02:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:02:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:02:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:02:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:02:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:02:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:02:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:02:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:02:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:02:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:02:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:02:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:02:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:02:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:02:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:02:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:02:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:02:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:02:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:02:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:02:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:02:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:02:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:02:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:02:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:02:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:02:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:02:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:02:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:02:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:02:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:02:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:02:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:02:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:02:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 197.65612899000001, 2024-08-30T16:02:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:02:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:02:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:02:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:02:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:02:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:02:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:02:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:02:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:02:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:02:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:02:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:02:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:02:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:02:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:02:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:02:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:02:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:02:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:02:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 197.379954924, 2024-08-30T16:02:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:02:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:02:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:02:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:02:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:02:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:02:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:02:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:02:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:02:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:02:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:02:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:02:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:02:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:02:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:02:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:02:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:02:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:02:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:02:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:02:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:02:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 196.721663345, 2024-08-30T16:02:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:02:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:02:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:02:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:02:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:02:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:02:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:02:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:02:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:02:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:02:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:02:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:02:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:02:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:02:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:02:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:02:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:02:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:02:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:02:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:02:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:02:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:02:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:02:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:02:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:02:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:02:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:02:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:02:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:02:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:02:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:02:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:02:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:02:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:02:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 196.62575006099999, 2024-08-30T16:02:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:02:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:02:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:02:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:02:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:02:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:02:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:02:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:02:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:02:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:02:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:02:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:02:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:02:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:02:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:02:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:02:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:02:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:02:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:02:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 137.37438954999999, 2024-08-30T16:02:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:02:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:02:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:02:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:02:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:02:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:02:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:02:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:02:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:02:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:02:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:02:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:02:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:02:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:02:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:02:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:02:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:02:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:02:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:02:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:02:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:02:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 77.365284862999999, 2024-08-30T16:02:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:02:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:02:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:02:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:02:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:02:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:02:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:02:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:02:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:02:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:02:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:02:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:02:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:02:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:02:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:02:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:02:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:02:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:02:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:02:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:44.395 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:44.397 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:44.398 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:44.396+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:44.498 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:44.496+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:44.527 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:44.528 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:02:44.528 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:44.537 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:44.599 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:44.597+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:44.659 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:44.659 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:44.659 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:02:44.659 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:02:44.659 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:44.667 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:44.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:02:44.710 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:44.710+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:44.790 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:44.790 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:02:44.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:02:44.790 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:02:44.790 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:44.799 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:44.800 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:44.799+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:44.901 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:44.899+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:44.989 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:44.989 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:02:44.989 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:45.002 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:45.002 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:45.000+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:45.103 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:45.102+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:45.128 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:45.128 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:45.129 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:02:45.129 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:02:45.129 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:45.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:02:45.138 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:45.205 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:45.204+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:45.269 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:07.874 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:45.305 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:45.305+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:45.316 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:45.317 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:02:45.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:02:45.317 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:02:45.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.317 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:02:45.317 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:02:45.317 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 214.450735326, 2024-08-30T16:02:45.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:02:45.317 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.317 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.317 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.318 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.318 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.318 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:02:45.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.318 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:02:45.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:02:45.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:02:45.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:02:45.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:02:45.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:02:45.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:02:45.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:02:45.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:02:45.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:02:45.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:02:45.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:02:45.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:02:45.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:02:45.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:02:45.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.321 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.321 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:02:45.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:02:45.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:02:45.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:02:45.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:02:45.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:02:45.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:02:45.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:02:45.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:02:45.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:02:45.323 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.323 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.323 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.324 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:02:45.324 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:02:45.324 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 213.53431930599999, 2024-08-30T16:02:45.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:02:45.324 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.324 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.324 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.324 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.324 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.324 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:02:45.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.324 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:02:45.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:02:45.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:02:45.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:02:45.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:02:45.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:02:45.326 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.326 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.326 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.326 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:02:45.326 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:02:45.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:02:45.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:02:45.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.327 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.327 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:02:45.327 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:02:45.327 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.327 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:02:45.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:02:45.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:02:45.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:02:45.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.328 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.328 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:02:45.328 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:02:45.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.329 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.329 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:02:45.329 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:02:45.329 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.329 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.329 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.330 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:02:45.330 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:02:45.330 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 213.53261475299999, 2024-08-30T16:02:45.330 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:02:45.330 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.330 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.330 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.330 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.330 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.330 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:02:45.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.331 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:02:45.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.331 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.331 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:02:45.331 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:02:45.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:02:45.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:02:45.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:02:45.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.332 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.332 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:02:45.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:02:45.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:02:45.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:02:45.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.333 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:02:45.333 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:02:45.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.333 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:02:45.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:02:45.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:02:45.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:02:45.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.334 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.334 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:02:45.334 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:02:45.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.335 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.335 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:02:45.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:02:45.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.335 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.335 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.335 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:02:45.335 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 213.11316094599999, 2024-08-30T16:02:45.335 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:02:45.335 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.336 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.336 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.336 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.336 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.336 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:02:45.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.336 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.336 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:02:45.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:02:45.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.337 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.337 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:02:45.337 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:02:45.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:02:45.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:02:45.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:02:45.338 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:02:45.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.338 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.338 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:02:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:02:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:02:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:02:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:02:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:02:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.339 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:02:45.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:02:45.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:02:45.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:02:45.340 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.340 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.340 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.340 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:02:45.340 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:02:45.340 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.341 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.341 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.341 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.341 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.341 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.341 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:02:45.341 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 213.09627277199999, 2024-08-30T16:02:45.341 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:02:45.341 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.341 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.341 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.342 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:02:45.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.342 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:02:45.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.342 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.342 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.342 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.342 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:02:45.342 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:02:45.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:02:45.344 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.344 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.344 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.344 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:02:45.344 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:02:45.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:02:45.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:02:45.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.345 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:02:45.345 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:02:45.345 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.345 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.345 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:02:45.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:02:45.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:02:45.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:02:45.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.346 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.346 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.346 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:02:45.346 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:02:45.346 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:02:45.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:02:45.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.347 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.347 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.347 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:02:45.347 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:02:45.347 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.347 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.347 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.347 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.348 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.348 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.348 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:02:45.348 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 213.08327270300001, 2024-08-30T16:02:45.348 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:02:45.348 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.348 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.348 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.348 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.348 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.348 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:02:45.348 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.349 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:02:45.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:02:45.349 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.349 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.349 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.349 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.349 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:02:45.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:02:45.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:02:45.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:02:45.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.350 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.350 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:02:45.350 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:02:45.350 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.350 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:02:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:02:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:02:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:02:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:02:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:02:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:02:45.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:02:45.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.352 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:02:45.352 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:02:45.352 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.352 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.352 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:02:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:02:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:02:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 213.08078041100001, 2024-08-30T16:02:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:02:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.353 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.354 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.354 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.354 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.354 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:02:45.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.354 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.354 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:02:45.354 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.354 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.354 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.354 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:02:45.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:02:45.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:02:45.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.355 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.355 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:02:45.355 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:02:45.355 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.355 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:02:45.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:02:45.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:02:45.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:02:45.356 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.356 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.356 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.356 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:02:45.356 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:02:45.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:02:45.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:02:45.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.357 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:02:45.357 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:02:45.357 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.357 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.357 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:02:45.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:02:45.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.358 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.358 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:02:45.358 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:02:45.358 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.358 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.358 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.358 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.358 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.358 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.358 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:02:45.359 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 213.080022963, 2024-08-30T16:02:45.359 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:02:45.359 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.359 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.359 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.359 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.359 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.359 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:02:45.359 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.359 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.359 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.359 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.359 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:02:45.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:02:45.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.360 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:02:45.360 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:02:45.360 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.360 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.360 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:02:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:02:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:02:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:02:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:02:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:02:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:02:45.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:02:45.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:02:45.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:02:45.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.362 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.362 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.362 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:02:45.362 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:02:45.362 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:02:45.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:02:45.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.363 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.363 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.363 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:02:45.363 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:02:45.363 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.363 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.363 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.363 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.364 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.364 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.364 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:02:45.364 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 213.07929051299999, 2024-08-30T16:02:45.364 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:02:45.364 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.364 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.364 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.364 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.364 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.364 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:02:45.364 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.364 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:02:45.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:02:45.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.365 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.365 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.365 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.365 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:02:45.365 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:02:45.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:02:45.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:02:45.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:02:45.366 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:02:45.366 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.366 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.366 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.366 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:02:45.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:02:45.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:02:45.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:02:45.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.367 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.367 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:02:45.367 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:02:45.367 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.367 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:02:45.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:02:45.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:02:45.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:02:45.368 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.368 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.368 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.368 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:02:45.368 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:02:45.368 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.369 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.369 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.369 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.369 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.369 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.369 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:02:45.369 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 213.07675171400001, 2024-08-30T16:02:45.369 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:02:45.369 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.369 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.369 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.369 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.369 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.370 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:02:45.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.370 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:02:45.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.370 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.370 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:02:45.370 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.370 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.370 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:02:45.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:02:45.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:02:45.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:02:45.371 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.371 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.371 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.371 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:02:45.371 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:02:45.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:02:45.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:02:45.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:02:45.372 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:02:45.372 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.372 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.372 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.372 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:02:45.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:02:45.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:02:45.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:02:45.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.373 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.373 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:02:45.373 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:02:45.373 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.373 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.374 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.374 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:02:45.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:02:45.374 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.374 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.374 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.374 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.374 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.374 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.374 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:02:45.374 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 213.07600371300001, 2024-08-30T16:02:45.374 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:02:45.374 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.375 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.375 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.375 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.375 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.375 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:02:45.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.375 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.375 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.375 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:02:45.375 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.375 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.375 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:02:45.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:02:45.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:02:45.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.376 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.376 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.376 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:02:45.376 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:02:45.376 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:02:45.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:02:45.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:02:45.377 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:02:45.377 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.377 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.377 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.377 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:02:45.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:02:45.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:02:45.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:02:45.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.378 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:02:45.378 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:02:45.378 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.378 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.378 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:02:45.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:02:45.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.379 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.379 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:02:45.379 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:02:45.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.379 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.379 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.379 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.379 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.379 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.380 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:02:45.380 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.30388724100001, 2024-08-30T16:02:45.380 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:02:45.380 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.380 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.380 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.380 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.380 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.380 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:02:45.380 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.380 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.380 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.380 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:02:45.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:02:45.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.381 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.381 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:02:45.381 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:02:45.381 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.381 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:02:45.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:02:45.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:02:45.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:02:45.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.382 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.382 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:02:45.382 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:02:45.382 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:02:45.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:02:45.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:02:45.383 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:02:45.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.383 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.383 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:02:45.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:02:45.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:02:45.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:02:45.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.384 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.384 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:02:45.384 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:02:45.384 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.385 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.385 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.385 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.385 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:02:45.385 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.30183009500001, 2024-08-30T16:02:45.385 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:02:45.385 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.385 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.385 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.385 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.385 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.386 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:02:45.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.386 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:02:45.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.386 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.386 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:02:45.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:02:45.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:02:45.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:02:45.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:02:45.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:02:45.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:02:45.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:02:45.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:02:45.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:02:45.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:02:45.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:02:45.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:02:45.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:02:45.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:02:45.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:02:45.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:02:45.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:02:45.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:02:45.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.390 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.390 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.390 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:02:45.390 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.03888559500001, 2024-08-30T16:02:45.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:02:45.391 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.391 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.391 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.391 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.391 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.391 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:02:45.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.391 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:02:45.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:02:45.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:02:45.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:02:45.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:02:45.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:02:45.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:02:45.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:02:45.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.393 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.393 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:02:45.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:02:45.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:02:45.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:02:45.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:02:45.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:02:45.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:02:45.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:02:45.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:02:45.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:02:45.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:02:45.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:02:45.395 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.395 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.395 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.396 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.396 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:02:45.396 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.03779409200001, 2024-08-30T16:02:45.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:02:45.396 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.396 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.396 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.396 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.396 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.396 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:02:45.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.397 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:02:45.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:02:45.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:02:45.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:02:45.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:02:45.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:02:45.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:02:45.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:02:45.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:02:45.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:02:45.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.399 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.399 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:02:45.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:02:45.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:02:45.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:02:45.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:02:45.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:02:45.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:02:45.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:02:45.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:02:45.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:02:45.401 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.401 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.401 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.401 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.401 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:02:45.401 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.03704450699999, 2024-08-30T16:02:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:02:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:02:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:02:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:02:45.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:02:45.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:02:45.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:02:45.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:02:45.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:02:45.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:02:45.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:02:45.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:02:45.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:02:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:02:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:02:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:02:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:02:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:02:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:02:45.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:02:45.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:02:45.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:02:45.406 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.406 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.407 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.407 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.407 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:02:45.407 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.003312975, 2024-08-30T16:02:45.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:02:45.407 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.407 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.407 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.407 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.407 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.407 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:02:45.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.408 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:02:45.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:02:45.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:02:45.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:02:45.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:02:45.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:02:45.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:02:45.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:02:45.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:02:45.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:02:45.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:02:45.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:02:45.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:02:45.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:02:45.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:02:45.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:02:45.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:02:45.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:02:45.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.412 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.412 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:02:45.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:02:45.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.412 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.412 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.412 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:02:45.412 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.00278696699999, 2024-08-30T16:02:45.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:02:45.412 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.413 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.413 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.413 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.413 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.413 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:02:45.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.413 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:02:45.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:02:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:02:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:02:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:02:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:02:45.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:02:45.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:02:45.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:02:45.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:02:45.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:02:45.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:02:45.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:02:45.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:02:45.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:02:45.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:02:45.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:02:45.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:02:45.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:02:45.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:02:45.417 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.417 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.417 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.417 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:02:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 212.00233140399999, 2024-08-30T16:02:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:02:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:02:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:02:45.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:02:45.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:02:45.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:02:45.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:02:45.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:02:45.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:02:45.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:02:45.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:02:45.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:02:45.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:02:45.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:02:45.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:02:45.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:02:45.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:02:45.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:02:45.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:02:45.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:02:45.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:02:45.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:02:45.422 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.423 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.423 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.423 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:02:45.423 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:02:45.423 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 207.33181608800001, 2024-08-30T16:02:45.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:02:45.423 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.423 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.423 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:45.423 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:45.423 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:45.424 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:02:45.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.424 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:02:45.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:02:45.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:02:45.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:02:45.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:02:45.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:02:45.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:02:45.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:02:45.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:02:45.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:02:45.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:02:45.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:02:45.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:02:45.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:02:45.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:45.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:02:45.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:02:45.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:02:45.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:02:45.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:02:45.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:02:45.428 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.428 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.428 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.428 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.428 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.428 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:45.430 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":38,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:02:45.430 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:45.431 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:45.429+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:45.431 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:02:45.431 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:02:45.491 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:08.096 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:45.524 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:02:45.524 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:02:45.530 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:45.529+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:45.596 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:45.596 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:45.596 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:02:45.596 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:45.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:45.615 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:02:45.615 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:02:45.642 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:45.641+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:45.704 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:02:45.704 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:02:45.732 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:45.731+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:45.797 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:45.797 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:45.797 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:02:45.797 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:02:45.798 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:45.809 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:45.823 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:02:45.823 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:02:45.833 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:45.831+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:45.907 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:02:45.908 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:02:45.933 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:45.933+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:45.950 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:45.950 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:02:45.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:02:45.950 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:02:45.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.950 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:02:45.950 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:02:45.950 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 199.42084696699999, 2024-08-30T16:02:45.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:02:45.951 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.951 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.951 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:02:45.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:02:45.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:02:45.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:02:45.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:02:45.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:02:45.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:02:45.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:02:45.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:02:45.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:02:45.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:02:45.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:02:45.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:02:45.954 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:02:45.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.954 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.954 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:02:45.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:02:45.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:02:45.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:02:45.957 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.957 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.957 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.957 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:02:45.957 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:02:45.957 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 198.302398821, 2024-08-30T16:02:45.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:02:45.957 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:45.957 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:45.957 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:45.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:45.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:02:45.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:45.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:02:45.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:45.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:45.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:02:45.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:02:45.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:45.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:02:45.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:02:45.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:45.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:02:45.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:02:45.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:45.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:02:45.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:02:45.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:45.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:02:45.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:02:45.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:45.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:02:45.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:02:45.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:45.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:02:45.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:02:45.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:45.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:45.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:45.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:02:45.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:02:45.962 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.962 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.962 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.962 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:45.962 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:45.962 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:45.963 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:46.006 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:02:46.007 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:02:46.034 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:46.033+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:46.090 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:02:46.090 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:02:46.129 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:46.129 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:46.138 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:46.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-08-30T16:02:46.143 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:46.141+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:46.236 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:46.235+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:46.263 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:46.263 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:46.272 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:46.337 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:46.335+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:46.401 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:46.401 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:46.411 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:46.452 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:46.450+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:46.538 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:46.537+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:46.556 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:46.556 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:46.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:46.640 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:46.638+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:46.714 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:46.714 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:46.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:46.740 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:46.739+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:46.842 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:46.840+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:46.903 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:46.903 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:46.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:46.954 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:46.953+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:47.043 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:47.042+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:47.083 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:47.083 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:47.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:47.144 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:47.142+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:47.245 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:47.243+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:47.260 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:47.261 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:47.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:47.346 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:47.345+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:47.400 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:47.401 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:47.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:47.447 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:47.446+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:47.548 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:47.546+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:47.576 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:47.576 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:47.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:47.648 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:47.647+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:47.712 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-08-30T16:02:47.718 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:47.718 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:47.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:47.774 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:47.772+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:47.851 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:47.849+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:47.858 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:47.858 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:47.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:47.952 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:47.951+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:48.040 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:48.041 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:48.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:48.090 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:48.089+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:48.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:02:48.144 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7] out_osds: [6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:02:48.144 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:02:48.144 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-08-30T16:02:48.145 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T16:02:48.145 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T16:02:48.145 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-08-30T16:02:48.177 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:48.175+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:48.219 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:48.219 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:48.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:48.270 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:48.269+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:48.274 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:48.274 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-30T16:02:48.274 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:48.356 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:48.354+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:48.367 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:02:48.368 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:02:48.398 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:48.399 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:48.401 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:11.007 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:48.408 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:48.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:02:48.477 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:02:48.478 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:02:48.478 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:48.476+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:48.481 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:11.087 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:48.532 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:48.533 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:48.542 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:48.558 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:48.556+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:48.601 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:02:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:02:48.659 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:48.658+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:48.668 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:48.668 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:48.678 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:48.706 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:02:48.707 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:02:48.760 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:48.759+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:48.761 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:11.367 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:02:48.792 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:02:48.793 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:02:48.827 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:02:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:02:48.828 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:02:48.828 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:48.837 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:48.876 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:02:48.876 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:02:48.882 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:48.880+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:48.961 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:02:48.961 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:02:48.962 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:48.960+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:48.979 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:48.979 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:48.979 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:02:48.979 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:48.990 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:49.057 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:02:49.058 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:02:49.081 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:49.079+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:49.157 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:02:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:02:49.157 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:49.163 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:49.161+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:49.166 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:49.264 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:49.262+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:49.385 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:49.386 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:02:49.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:02:49.386 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:02:49.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.386 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:02:49.386 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:02:49.386 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 218.52255283100001, 2024-08-30T16:02:49.386 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:02:49.386 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.386 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.386 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.386 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.386 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:02:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:02:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:02:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:02:49.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:02:49.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.388 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:02:49.388 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:02:49.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.388 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:02:49.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:02:49.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.389 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.389 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:02:49.389 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:02:49.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:02:49.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:02:49.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:02:49.390 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:02:49.390 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:02:49.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.390 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:02:49.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:02:49.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:02:49.391 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:02:49.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.391 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.391 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:02:49.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:02:49.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:02:49.392 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:02:49.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.392 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.392 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:02:49.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:02:49.393 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.393 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.393 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.393 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:02:49.393 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:02:49.393 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.606136811, 2024-08-30T16:02:49.393 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:02:49.394 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.394 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.394 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.394 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.394 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.394 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:02:49.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.394 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.394 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.394 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.394 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:02:49.394 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.394 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:02:49.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:02:49.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:02:49.395 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.395 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.395 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.395 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:02:49.395 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:02:49.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:02:49.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:02:49.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.396 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:02:49.396 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:02:49.396 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.396 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.396 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:02:49.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:02:49.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:02:49.397 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:02:49.397 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.397 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.397 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.397 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:02:49.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:02:49.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:02:49.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:02:49.398 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.398 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.398 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.398 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:02:49.398 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:02:49.398 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.398 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.398 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.399 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.399 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:02:49.399 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:02:49.399 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.604432258, 2024-08-30T16:02:49.399 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:02:49.399 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.399 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.399 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.399 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.399 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.399 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:02:49.399 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.400 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:02:49.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.400 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.400 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.400 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:02:49.400 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.400 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:02:49.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:02:49.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.401 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.401 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.401 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:02:49.401 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:02:49.401 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:02:49.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:02:49.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.402 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.402 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.402 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:02:49.402 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:02:49.402 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:02:49.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:02:49.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.403 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.403 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.403 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:02:49.403 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:02:49.403 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:02:49.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:02:49.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:02:49.404 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:02:49.404 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.404 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.404 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.404 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:02:49.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:02:49.405 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.405 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.405 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.405 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.405 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.405 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.405 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:02:49.405 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.18497845100001, 2024-08-30T16:02:49.405 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:02:49.405 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.405 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.406 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.406 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:02:49.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.406 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.406 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.406 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:02:49.406 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.406 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.406 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:02:49.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.407 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:02:49.407 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:02:49.407 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.407 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.407 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:02:49.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:02:49.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:02:49.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:02:49.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.408 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.408 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.408 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:02:49.408 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:02:49.408 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:02:49.409 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:02:49.409 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.409 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.409 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.409 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:02:49.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:02:49.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:02:49.410 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:02:49.410 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.410 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.410 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.410 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:02:49.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:02:49.411 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.411 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.411 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.411 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:02:49.411 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:02:49.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.412 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.412 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.412 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.412 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.413 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:02:49.413 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.168090277, 2024-08-30T16:02:49.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:02:49.413 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.413 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.413 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.413 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.413 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.414 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:02:49.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.414 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:02:49.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:02:49.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:02:49.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:02:49.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:02:49.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:02:49.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:02:49.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:02:49.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:02:49.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:02:49.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:02:49.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:02:49.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.419 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:02:49.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:02:49.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.419 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:02:49.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:02:49.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:02:49.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:02:49.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:02:49.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:02:49.421 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.421 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.421 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.422 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.422 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:02:49.422 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.15509020799999, 2024-08-30T16:02:49.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:02:49.422 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.422 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.422 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.422 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.423 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.423 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:02:49.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.423 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:02:49.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:02:49.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:02:49.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:02:49.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.425 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.425 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:02:49.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:02:49.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:02:49.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:02:49.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:02:49.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:02:49.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:02:49.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:02:49.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:02:49.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:02:49.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:02:49.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:02:49.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:02:49.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:02:49.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:02:49.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:02:49.429 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.429 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.429 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.430 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.430 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:02:49.430 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.15259791599999, 2024-08-30T16:02:49.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:02:49.430 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.430 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.430 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.430 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:02:49.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.430 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:02:49.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:02:49.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:02:49.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:02:49.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:02:49.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:02:49.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:02:49.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:02:49.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:02:49.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:02:49.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:02:49.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:02:49.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:02:49.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:02:49.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:02:49.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:02:49.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:02:49.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:02:49.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:02:49.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:02:49.435 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.435 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.435 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.435 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.435 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:02:49.435 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.15184046799999, 2024-08-30T16:02:49.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:02:49.435 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.436 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.436 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.436 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.436 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.436 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:02:49.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.436 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.436 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.436 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:02:49.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:02:49.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:02:49.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:02:49.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:02:49.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:02:49.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:02:49.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:02:49.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:02:49.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:02:49.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:02:49.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:02:49.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:02:49.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:02:49.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:02:49.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:02:49.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:02:49.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:02:49.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:02:49.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:02:49.440 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.440 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.440 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.441 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.441 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:02:49.441 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.151108018, 2024-08-30T16:02:49.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:02:49.441 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.441 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.441 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.441 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.441 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.442 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:02:49.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.442 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:02:49.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.442 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:02:49.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:02:49.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:02:49.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:02:49.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:02:49.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:02:49.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:02:49.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:02:49.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:02:49.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:02:49.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:02:49.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:02:49.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:02:49.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:02:49.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:02:49.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:02:49.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:02:49.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:02:49.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:02:49.446 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.446 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.446 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.447 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.447 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:02:49.447 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.148569219, 2024-08-30T16:02:49.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:02:49.447 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.447 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.447 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.447 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.447 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.447 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:02:49.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.447 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:02:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:02:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:02:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:02:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:02:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:02:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:02:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:02:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:02:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:02:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:02:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:02:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:02:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:02:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:02:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:02:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:02:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:02:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:02:49.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:02:49.452 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.452 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.452 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.452 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.452 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:02:49.452 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 217.14782121799999, 2024-08-30T16:02:49.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:02:49.452 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.452 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.452 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.453 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.453 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.453 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:02:49.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.453 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:02:49.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:02:49.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:02:49.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:02:49.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:02:49.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:02:49.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:02:49.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:02:49.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:02:49.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:02:49.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:02:49.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:02:49.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:02:49.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:02:49.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:02:49.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:02:49.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:02:49.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:02:49.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:02:49.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:02:49.461 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.461 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.462 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.462 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.462 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:02:49.462 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 216.375704746, 2024-08-30T16:02:49.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:02:49.463 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.463 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.463 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.463 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.463 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.464 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:02:49.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.464 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:02:49.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:02:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:02:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:02:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:02:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:02:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:02:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:02:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:02:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:02:49.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:02:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:02:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:02:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:02:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:02:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:02:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:02:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:02:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:02:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:02:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:02:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 216.3736476, 2024-08-30T16:02:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:02:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.469 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:02:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:02:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:02:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:02:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:02:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:02:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:02:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:02:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:02:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:02:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:02:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:02:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:02:49.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:02:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:02:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:02:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:02:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:02:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:02:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:02:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:02:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:02:49.474 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 216.11070309999999, 2024-08-30T16:02:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:02:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:02:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:02:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:02:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:02:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:02:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:02:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:02:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:02:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:02:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:02:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:02:49.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:02:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:02:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:02:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:02:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:02:49.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:02:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:02:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:02:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:02:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:02:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.479 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:02:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 216.109611597, 2024-08-30T16:02:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:02:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:02:49.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:02:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:02:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:02:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:02:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:02:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:02:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:02:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:02:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:02:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:02:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:02:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:02:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:02:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:02:49.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:02:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:02:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:02:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:02:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:02:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:02:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:02:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 216.108862012, 2024-08-30T16:02:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:02:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:02:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:02:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:02:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:02:49.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:02:49.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:02:49.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:02:49.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:02:49.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:02:49.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:02:49.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:02:49.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:02:49.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:02:49.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:02:49.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:02:49.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:02:49.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:02:49.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:02:49.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:02:49.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:02:49.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:02:49.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.490 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.490 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.490 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.490 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:02:49.490 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 216.07513048000001, 2024-08-30T16:02:49.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:02:49.490 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.491 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.491 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.491 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.491 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.491 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:02:49.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.491 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:02:49.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:02:49.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:02:49.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:02:49.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:02:49.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:02:49.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:02:49.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:02:49.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:02:49.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:02:49.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:02:49.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:02:49.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:02:49.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:02:49.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:02:49.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:02:49.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:02:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:02:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:02:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:02:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.495 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:02:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 216.074604472, 2024-08-30T16:02:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:02:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:02:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:02:49.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:02:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:02:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:02:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:02:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:02:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:02:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:02:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:02:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:02:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:02:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:02:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:02:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:02:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:02:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:02:49.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:02:49.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:02:49.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:02:49.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:02:49.500 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.500 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.500 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.501 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:49.501 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:02:49.501 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 216.074148909, 2024-08-30T16:02:49.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:02:49.501 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.501 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.501 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.501 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.501 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.501 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:02:49.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.501 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:02:49.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:02:49.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:02:49.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:02:49.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:02:49.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:02:49.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:02:49.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:02:49.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:02:49.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:02:49.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:02:49.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:02:49.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:02:49.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:02:49.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:02:49.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:02:49.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:02:49.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:02:49.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:02:49.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:02:49.506 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.506 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.506 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.506 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:02:49.506 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:02:49.506 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 211.403633593, 2024-08-30T16:02:49.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:02:49.506 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:49.506 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:49.506 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:49.506 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:49.507 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:49.507 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:02:49.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.507 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:49.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:49.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:02:49.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:49.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:02:49.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:02:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:02:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:02:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:02:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:49.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:02:49.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:02:49.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:49.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:02:49.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:02:49.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:49.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:02:49.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:02:49.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:49.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:02:49.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:02:49.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:49.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:02:49.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:02:49.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:49.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:02:49.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:02:49.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:49.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:49.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:49.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:02:49.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:02:49.511 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.511 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.511 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.511 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:49.511 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:49.511 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:49.512 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:49.386+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:49.513 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:02:49.514 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-08-30T16:02:49.514 INFO:tasks.mon_recovery:removing mon b 2024-08-30T16:02:49.514 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-30T16:02:49.514 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:02:49.514 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:49.515 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:49.511+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:49.570 INFO:tasks.ceph.mon.b.smithi094.stderr:2024-08-30T16:02:49.556+0000 a084640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 36139) UID: 0 2024-08-30T16:02:49.573 INFO:tasks.ceph.mon.b.smithi094.stderr:2024-08-30T16:02:49.558+0000 a084640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-08-30T16:02:49.613 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:49.612+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:49.698 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:49.698 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:49.698 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:02:49.698 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:49.708 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:49.716 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:49.714+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:49.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-08-30T16:02:49.836 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:49.834+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:49.878 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:49.878 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:49.879 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:02:49.879 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:02:49.879 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:49.889 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:49.933 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:49.931+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:50.017 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:50.016+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:50.018 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:50.018 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:02:50.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:02:50.018 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:02:50.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.018 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:02:50.018 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:02:50.018 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 203.48813728600001, 2024-08-30T16:02:50.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:02:50.018 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:50.018 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:02:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:02:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:50.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:02:50.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:02:50.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:50.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:02:50.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:02:50.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:50.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:02:50.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:02:50.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:50.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:02:50.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:02:50.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:50.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:02:50.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:02:50.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:50.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:02:50.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:02:50.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:50.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:02:50.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:02:50.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:50.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:02:50.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:02:50.022 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:50.023 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:50.023 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:50.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.023 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:02:50.023 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:02:50.023 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 202.36968913999999, 2024-08-30T16:02:50.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:02:50.023 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:50.023 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:50.023 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:50.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:50.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:02:50.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:50.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:50.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:02:50.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:50.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:50.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:02:50.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:02:50.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:50.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:02:50.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:02:50.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:50.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:02:50.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:02:50.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:50.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:02:50.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:02:50.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:50.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:02:50.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:02:50.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:50.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:02:50.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:02:50.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:50.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:02:50.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:02:50.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:50.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:02:50.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:02:50.027 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:50.027 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:50.028 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:50.028 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:50.028 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:50.028 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:50.030 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:50.117 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:50.116+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:50.214 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:50.214 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:50.214 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:02:50.214 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:50.218 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:50.216+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:50.223 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:50.319 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:50.317+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:50.388 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:02:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:02:50.388 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:50.397 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:50.439 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:50.437+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:50.520 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:50.519+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:50.565 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:50.565 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:02:50.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:02:50.565 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:02:50.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.565 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:02:50.565 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:02:50.565 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 202.185337997, 2024-08-30T16:02:50.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:02:50.566 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:50.566 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:02:50.566 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:50.566 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:02:50.566 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:02:50.566 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:02:50.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.566 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:50.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:50.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:02:50.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:50.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:50.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:02:50.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:50.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:50.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:02:50.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:02:50.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:50.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:02:50.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:02:50.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:50.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:02:50.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:02:50.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:50.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:02:50.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:02:50.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:50.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:02:50.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:02:50.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:50.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:02:50.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:02:50.569 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:50.569 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:50.569 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:50.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.569 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:02:50.570 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:02:50.570 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 82.072341881, 2024-08-30T16:02:50.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:02:50.570 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:50.570 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:02:50.570 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:50.570 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:02:50.570 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:02:50.570 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:02:50.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.570 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:50.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:50.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:02:50.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:50.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:50.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:02:50.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:50.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:50.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:02:50.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:02:50.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:50.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:02:50.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:02:50.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:50.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:02:50.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:02:50.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:50.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:02:50.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:02:50.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:50.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:02:50.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:02:50.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:50.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:50.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:50.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:02:50.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:02:50.573 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:50.573 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:50.573 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:50.573 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:50.574 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:50.574 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:50.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:50.621 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:50.619+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:50.703 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:50.703 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:50.703 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:50.703 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:50.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:50.758 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:50.756+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:50.822 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:50.821+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:50.839 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:50.839 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:50.839 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:50.839 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:50.840 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:50.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:50.923 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:50.921+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:51.020 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:51.020 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:02:51.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:02:51.020 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:02:51.020 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:51.024 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:51.022+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:51.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:51.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:02:51.143 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:51.142+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:51.157 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:51.157 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:51.157 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:51.157 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:51.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:51.226 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:51.224+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:51.326 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:51.324+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:51.336 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:51.337 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:51.337 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:51.337 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:51.337 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:51.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:51.429 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:51.428+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:51.525 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:02:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:02:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:02:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:02:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 210.250967024, 2024-08-30T16:02:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:02:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:02:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:02:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:02:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:02:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:02:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:02:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:02:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:02:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:02:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:02:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:02:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:02:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:02:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:02:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:02:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 209.978039663, 2024-08-30T16:02:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:02:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:02:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:02:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:02:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:02:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:02:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:02:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:02:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:02:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:02:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:02:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:02:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:02:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:02:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:02:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:51.534 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:51.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:51.536 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:51.534+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:51.553 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:02:51.553 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:02:51.661 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:51.659+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:51.661 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:02:51.661 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:02:51.693 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:51.693 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:51.693 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:51.693 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:51.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:51.738 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:51.736+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:51.764 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:02:51.764 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:02:51.855 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:02:51.855 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:02:51.856 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:51.854+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:51.876 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:51.876 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:51.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:51.937 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:02:51.937 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:02:51.940 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:51.938+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:52.021 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:02:52.021 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:02:52.041 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:52.039+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:52.053 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:52.054 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:52.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:52.101 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:02:52.102 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:02:52.145 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:52.144+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:52.182 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:02:52.182 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:02:52.230 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:52.230 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:52.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:52.281 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:52.280+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:52.343 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:52.342+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:52.409 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:52.409 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:52.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:52.444 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:52.442+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:52.545 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:52.544+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:52.586 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:52.586 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:52.595 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:52.647 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:52.645+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:52.722 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:52.722 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:52.730 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:52.771 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:52.769+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:52.849 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:52.847+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:52.894 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:52.894 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:52.904 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:52.988 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:52.986+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:53.051 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:53.050+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:53.073 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:53.073 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:53.083 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:53.152 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:53.151+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:53.247 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:53.247 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:53.253 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:53.251+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:53.256 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:53.283 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7] out_osds: [6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:02:53.284 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:02:53.284 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 5, 7] 2024-08-30T16:02:53.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2024-08-30T16:02:53.372 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:53.370+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:53.383 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:53.383 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:53.394 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:53.478 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:53.476+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:53.556 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:53.554+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:53.557 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:53.557 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:53.566 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:53.657 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:53.655+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:53.733 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:53.733 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:53.742 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:53.784 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:53.782+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:53.858 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:53.856+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:53.905 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:53.905 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:53.913 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:53.939 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T16:02:53.923+0000 1008f640 -1 mon.a@0(leader).osd e63 definitely_dead 0 2024-08-30T16:02:53.958 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:53.956+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:54.036 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:54.036 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:54.044 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:54.059 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:54.058+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:54.160 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:54.159+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:54.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:02:54.209 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:54.209 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:54.218 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:54.271 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:54.269+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:54.362 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:54.361+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:54.398 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:54.398 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:54.408 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:54.432 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:02:54.432 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:02:54.463 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:54.461+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:54.522 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:02:54.522 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:02:54.532 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:02:54.532 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:02:54.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:54.584 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:54.582+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:54.614 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:02:54.614 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:02:54.665 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:54.663+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:54.684 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:54.685 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:54.685 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:54.685 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:54.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:54.722 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:02:54.723 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:02:54.769 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:54.767+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:54.804 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:02:54.804 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:02:54.850 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:54.850 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:54.850 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:54.851 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:54.851 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:54.859 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:54.866 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:54.865+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:54.886 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:02:54.886 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:02:54.966 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:02:54.966 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:02:54.967 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:54.966+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:55.019 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:55.020 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:02:55.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:02:55.020 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:02:55.020 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:55.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:55.056 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:02:55.056 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:02:55.068 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:55.067+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:55.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-08-30T16:02:55.169 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:55.167+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:55.184 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:55.184 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:55.184 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:55.184 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:55.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:55.269 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:55.268+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:55.324 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:55.324 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:55.324 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:55.324 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:55.325 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:55.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:55.377 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:55.376+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:55.468 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:55.468 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:02:55.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:02:55.468 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:02:55.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.469 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:02:55.469 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:55.469 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 214.193825191, 2024-08-30T16:02:55.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:02:55.469 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:55.469 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:55.469 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:55.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:55.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:55.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:55.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:55.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:55.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:55.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:55.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:02:55.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:02:55.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:55.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:02:55.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:02:55.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:55.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:02:55.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:02:55.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:55.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:02:55.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:02:55.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:55.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:02:55.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:02:55.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:55.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:02:55.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:02:55.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:55.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:02:55.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:02:55.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:02:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 213.92089783, 2024-08-30T16:02:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:02:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:55.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:55.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:55.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:55.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:55.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:55.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:02:55.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:02:55.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:55.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:02:55.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:02:55.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:55.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:02:55.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:02:55.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:55.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:02:55.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:02:55.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:55.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:02:55.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:02:55.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:55.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:02:55.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:02:55.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:55.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:02:55.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:02:55.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:55.477 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:55.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:55.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:55.477 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:55.477 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:55.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:55.479 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:55.477+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:55.580 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:55.579+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:55.615 INFO:tasks.ceph.mon.b:Stopped 2024-08-30T16:02:55.616 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2024-08-30T16:02:55.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:02:55.650 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:55.651 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:55.651 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:55.651 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:55.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:55.702 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:55.701+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:55.782 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:55.781+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:55.828 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:55.829 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:55.829 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:55.829 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:55.829 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:55.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:55.883 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:55.882+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:55.976 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:55.976 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:02:55.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:02:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:02:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:02:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:02:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 210.599094772, 2024-08-30T16:02:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:02:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:02:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:02:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:02:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:02:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:02:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:02:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:02:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:02:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:02:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:02:55.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:02:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:02:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:02:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:02:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:02:55.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:02:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:02:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:02:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:02:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:55.981 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:55.981 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:55.984 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:55.982+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:55.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:02:56.085 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:56.084+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:56.154 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:56.154 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:56.154 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:56.154 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:56.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:02:56.205 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:56.203+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:56.287 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:56.285+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:56.331 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:56.331 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:56.331 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:56.332 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:56.332 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:56.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:02:56.425 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:56.423+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:56.488 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:56.487+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:56.533 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:02:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:02:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:02:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:02:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:02:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 212.08029127200001, 2024-08-30T16:02:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:02:56.533 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:02:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:02:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:02:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:02:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:02:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:02:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:02:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:02:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:02:56.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:02:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:02:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:02:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:02:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:02:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:02:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:02:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:02:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:02:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:02:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:02:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:56.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:02:56.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:02:56.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:02:56.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:02:56.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:02:56.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:02:56.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:02:56.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:02:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:02:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:02:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:02:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:02:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:02:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:02:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:56.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:02:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:02:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:02:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:02:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 210.96442493699999, 2024-08-30T16:02:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:02:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:56.540 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:02:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:02:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:02:56.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:02:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:02:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:02:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:02:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:02:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:02:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:02:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:02:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:02:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:02:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:02:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:02:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:02:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:02:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:02:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 210.90483343899999, 2024-08-30T16:02:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:02:56.544 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:02:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:02:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:02:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:02:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:56.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:02:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:02:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:02:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:02:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:02:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:02:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:02:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:02:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:02:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:02:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:02:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:02:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:02:56.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:02:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:02:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:02:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 210.87813475199999, 2024-08-30T16:02:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:02:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:02:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:56.549 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:02:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:02:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:02:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:02:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:02:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:02:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:02:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:02:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:02:56.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:02:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:02:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:02:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:02:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:56.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:02:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:02:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:02:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:02:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:02:56.553 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:02:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 210.01051822400001, 2024-08-30T16:02:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:02:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:02:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:02:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:02:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:02:56.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:02:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:02:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:02:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:02:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:02:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:02:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:02:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:02:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:02:56.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:02:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:02:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:02:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:02:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:02:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:02:56.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:02:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:02:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:02:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:02:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:02:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:02:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:02:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:02:56.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:02:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:02:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:02:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:02:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:02:56.559 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:02:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 209.851271145, 2024-08-30T16:02:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:02:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:02:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:56.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:02:56.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:56.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:02:56.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:02:56.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:02:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:02:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:02:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:02:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:02:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:02:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:02:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:02:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:02:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:02:56.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:02:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:02:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.565 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:02:56.565 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:02:56.565 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 209.57509707899999, 2024-08-30T16:02:56.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:02:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:02:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:02:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:02:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:02:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:02:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:02:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:02:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:02:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:56.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:02:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:02:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:02:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:02:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:02:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:02:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:02:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:02:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:02:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:02:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.569 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:02:56.570 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:02:56.570 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 208.91680550000001, 2024-08-30T16:02:56.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:02:56.570 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:56.570 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:02:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:02:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:02:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:02:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:02:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:02:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:02:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:02:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:02:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:02:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:02:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:02:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:02:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:02:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:02:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:02:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:02:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:02:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:02:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:02:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:02:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:02:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:02:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:02:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:02:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:02:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:02:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:02:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:02:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:02:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:02:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.577 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:02:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:02:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 208.820892216, 2024-08-30T16:02:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:02:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:56.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:02:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:02:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:56.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:02:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:02:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:02:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:02:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:02:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:02:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:02:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:02:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:02:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:02:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:02:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:02:56.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:02:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:02:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:02:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:02:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 149.569531705, 2024-08-30T16:02:56.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:02:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:02:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:02:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:02:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:02:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:56.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:02:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:02:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:02:56.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:02:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:02:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:02:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:02:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:02:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:56.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:02:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:02:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:02:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:02:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:02:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:02:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.586 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:02:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:02:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 89.560427017999999, 2024-08-30T16:02:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:02:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:02:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:02:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:02:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:02:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:02:56.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:02:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:02:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:56.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:02:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:02:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:02:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:02:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:02:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:02:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:02:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:02:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:02:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:02:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:02:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:02:56.590 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:56.591 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:56.591 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:56.592 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:02:56.593 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:56.591+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:56.715 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:56.713+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:56.729 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:56.729 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:02:56.729 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:56.738 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:02:56.795 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:56.794+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:56.896 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:56.894+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:56.901 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:56.901 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:56.901 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:02:56.901 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:02:56.901 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:56.910 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:02:56.996 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:56.995+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:57.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:02:57.075 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:57.075 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:02:57.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:02:57.075 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:02:57.075 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:57.084 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:02:57.126 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:57.124+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:57.198 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:57.197+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:57.248 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:02:57.248 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:57.257 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:02:57.301 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:57.299+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:57.380 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:57.380 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:57.381 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:02:57.381 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:02:57.381 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:57.390 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:02:57.400 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:57.399+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:57.501 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:57.500+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:57.602 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:57.602 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:02:57.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:02:57.603 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:02:57.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.603 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:02:57.603 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:02:57.603 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 226.74113526799999, 2024-08-30T16:02:57.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:02:57.603 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.603 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.603 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.603 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.603 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.603 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:02:57.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.604 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:02:57.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:02:57.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:02:57.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:02:57.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:02:57.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:02:57.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:02:57.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:02:57.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:02:57.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:02:57.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:02:57.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:02:57.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:02:57.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:02:57.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:02:57.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:02:57.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:02:57.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:02:57.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:02:57.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:02:57.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:02:57.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:02:57.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:02:57.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:02:57.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:02:57.609 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.610 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.610 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.610 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:02:57.610 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:02:57.610 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 225.82471924800001, 2024-08-30T16:02:57.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:02:57.610 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.610 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.610 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.610 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.610 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.611 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:02:57.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.611 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.611 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:02:57.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:02:57.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:02:57.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:02:57.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:02:57.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:02:57.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:02:57.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:02:57.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:02:57.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:02:57.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:02:57.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:02:57.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:02:57.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:02:57.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:02:57.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:02:57.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:02:57.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:02:57.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.616 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.616 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:02:57.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:02:57.616 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.616 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.616 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.617 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:02:57.617 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:02:57.617 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 225.82301469500001, 2024-08-30T16:02:57.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:02:57.617 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.617 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.617 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.617 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.617 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.617 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:02:57.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.618 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:02:57.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:02:57.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:02:57.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:02:57.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:02:57.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:02:57.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:02:57.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:02:57.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:02:57.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:02:57.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:02:57.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:02:57.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:02:57.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:02:57.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.621 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:02:57.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:02:57.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:02:57.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:02:57.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:02:57.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:02:57.622 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.623 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.623 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.623 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.623 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:02:57.623 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 225.40356088799999, 2024-08-30T16:02:57.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:02:57.623 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.623 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.624 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.624 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.624 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.624 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:02:57.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.624 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:02:57.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:02:57.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:02:57.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:02:57.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:02:57.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:02:57.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:02:57.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:02:57.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:02:57.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:02:57.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:02:57.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:02:57.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:02:57.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:02:57.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:02:57.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:02:57.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:02:57.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:02:57.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:02:57.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:02:57.629 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.629 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.630 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.630 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.630 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:02:57.630 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 225.38667271400001, 2024-08-30T16:02:57.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:02:57.630 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.630 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.630 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.631 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.631 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:02:57.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.631 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:02:57.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:02:57.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:02:57.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:02:57.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.637 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:02:57.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:02:57.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.637 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:02:57.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:02:57.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:02:57.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:02:57.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:02:57.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:02:57.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:02:57.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:02:57.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:02:57.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:02:57.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:02:57.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:02:57.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:02:57.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:02:57.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.641 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.641 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.641 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:02:57.641 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 225.373672645, 2024-08-30T16:02:57.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:02:57.641 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.641 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.642 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.642 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.642 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.642 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:02:57.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.642 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:02:57.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:02:57.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:02:57.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:02:57.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:02:57.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:02:57.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:02:57.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:02:57.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:02:57.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:02:57.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:02:57.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:02:57.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:02:57.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:02:57.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:02:57.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:02:57.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:02:57.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:02:57.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:02:57.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:02:57.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.646 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.647 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.647 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:02:57.647 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 225.371180353, 2024-08-30T16:02:57.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:02:57.647 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.648 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.648 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.648 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.648 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.648 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:02:57.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.648 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:02:57.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:02:57.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:02:57.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:02:57.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:02:57.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:02:57.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:02:57.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:02:57.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:02:57.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:02:57.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:02:57.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:02:57.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:02:57.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:02:57.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:02:57.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:02:57.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:02:57.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:02:57.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:02:57.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:02:57.653 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.653 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.654 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.654 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.654 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:02:57.654 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 225.370422905, 2024-08-30T16:02:57.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:02:57.654 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.654 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.654 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.655 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:02:57.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.655 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:02:57.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.655 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.655 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:02:57.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:02:57.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:02:57.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:02:57.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:02:57.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:02:57.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:02:57.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:02:57.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:02:57.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.659 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:02:57.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:02:57.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.659 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:02:57.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:02:57.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:02:57.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:02:57.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:02:57.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:02:57.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:02:57.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:02:57.662 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.663 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.663 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.663 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.663 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:02:57.663 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 225.36969045500001, 2024-08-30T16:02:57.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:02:57.663 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.663 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.663 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.663 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.663 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.664 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:02:57.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.664 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:02:57.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:02:57.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:02:57.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:02:57.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:02:57.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:02:57.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:02:57.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:02:57.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:02:57.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:02:57.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:02:57.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:02:57.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:02:57.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:02:57.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:02:57.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:02:57.668 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.668 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.668 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.668 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:02:57.668 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:02:57.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.669 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.669 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:02:57.669 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:02:57.669 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.669 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.669 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.669 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.669 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.670 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.670 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:02:57.670 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 225.367151656, 2024-08-30T16:02:57.670 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:02:57.670 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.670 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.670 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.670 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:02:57.670 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.670 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.670 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:02:57.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:02:57.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.671 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.671 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.671 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.671 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:02:57.671 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:02:57.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:02:57.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:02:57.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.672 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.672 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:02:57.672 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:02:57.672 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.672 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:02:57.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:02:57.673 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.673 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.673 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.673 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:02:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:02:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:02:57.674 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:02:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:02:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:02:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:02:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:02:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:02:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:02:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:02:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 225.366403655, 2024-08-30T16:02:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:02:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.677 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:02:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:02:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:02:57.678 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:02:57.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:02:57.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.679 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:02:57.679 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:02:57.679 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.679 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.679 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:02:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:02:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:02:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:02:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.680 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:02:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:02:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:02:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:02:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:02:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:02:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.681 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:02:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:02:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:02:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:02:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:02:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.59428718300001, 2024-08-30T16:02:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:02:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.683 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:02:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:02:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:02:57.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:02:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:02:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:02:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:02:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:02:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:02:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:02:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:02:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:02:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:02:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.686 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:02:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:02:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:02:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:02:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:02:57.687 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:02:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:02:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:02:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:02:57.688 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.59223003700001, 2024-08-30T16:02:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:02:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:02:57.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:02:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:02:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:02:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:02:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:02:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:02:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:02:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:02:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:02:57.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:02:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:02:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:02:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:02:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:02:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:02:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:02:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:02:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:02:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:02:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:02:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:02:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.329285537, 2024-08-30T16:02:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:02:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.697 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:02:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:02:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:02:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:02:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:02:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:02:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:02:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:02:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:02:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:02:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:02:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:02:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:02:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:02:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:02:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:02:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:02:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:02:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:02:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:02:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:02:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.702 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:02:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.32819403400001, 2024-08-30T16:02:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:02:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.703 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:02:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:02:57.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:02:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:02:57.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:02:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:02:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:02:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:02:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:02:57.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:02:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:02:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:02:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:02:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:02:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:02:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:02:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:02:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:02:57.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:02:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:02:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:02:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.709 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:02:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.32744444900001, 2024-08-30T16:02:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:02:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:02:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:02:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:02:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:02:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:02:57.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:02:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:02:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:02:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:02:57.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:02:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:02:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:02:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:02:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:02:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:02:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:02:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:02:57.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:02:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:02:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:02:57.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:02:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:02:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.29371291699999, 2024-08-30T16:02:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:02:57.718 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:02:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:02:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:02:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:02:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:02:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:02:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:02:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:02:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:02:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:02:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:02:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:02:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:02:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:02:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:02:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:02:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:02:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:02:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:02:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:02:57.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:02:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:02:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.29318690900001, 2024-08-30T16:02:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:02:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.724 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:02:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:02:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:02:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:02:57.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:02:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:02:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:02:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:02:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:02:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:02:57.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:02:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:02:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:02:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:02:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:02:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:02:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:02:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:02:57.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:02:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:02:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:02:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.732 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:02:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:02:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.29273134600001, 2024-08-30T16:02:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:02:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.733 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:02:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:02:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:02:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:02:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:02:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:02:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:02:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:02:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:02:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:02:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:02:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:02:57.737 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:02:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:02:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 219.62221603, 2024-08-30T16:02:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:02:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:02:57.738 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:02:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:02:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:02:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:02:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:57.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:02:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:02:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:02:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:02:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:02:57.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:02:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:02:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:02:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:02:57.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:02:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:02:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:02:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:02:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:02:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:02:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:02:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:02:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:02:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:02:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:57.743 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:57.743 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:57.744 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:57.600+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:57.745 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:02:57.786 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:57.784+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:57.847 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:57.845+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:57.921 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:57.921 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:57.921 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:02:57.921 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:57.930 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:02:57.973 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:57.970+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:58.049 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:58.047+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:58.096 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:58.096 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:02:58.097 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:02:58.097 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:58.105 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:02:58.150 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:58.148+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:58.232 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:58.232 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:02:58.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:02:58.232 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:02:58.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.232 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:02:58.232 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:02:58.232 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 211.70220272200001, 2024-08-30T16:02:58.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:02:58.233 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:58.233 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:58.233 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:58.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:58.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:02:58.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:58.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:58.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:02:58.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:58.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:58.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:02:58.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:02:58.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:58.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:02:58.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:02:58.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:58.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:02:58.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:02:58.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:58.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:02:58.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:02:58.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:58.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:02:58.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:02:58.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:58.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:02:58.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:02:58.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:58.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:02:58.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:02:58.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:02:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:02:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:02:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:02:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 210.58375457599999, 2024-08-30T16:02:58.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:02:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:02:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:02:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:02:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:58.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:02:58.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:58.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:02:58.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:02:58.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:58.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:02:58.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:02:58.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:02:58.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:02:58.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:02:58.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:02:58.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:02:58.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:02:58.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:58.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:02:58.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:02:58.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:58.242 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:58.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:58.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:58.242 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:58.242 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:58.243 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:02:58.251 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:58.249+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:58.352 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:58.350+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:58.406 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:58.406 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:58.406 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:02:58.407 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:58.415 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:02:58.454 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:02:58.451+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:02:58.554 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:58.552+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:58.580 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:58.580 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:02:58.580 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:02:58.580 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:02:58.581 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:58.589 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:02:58.655 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:02:58.653+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:02:58.718 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:02:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:02:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:02:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:02:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:02:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:02:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 210.338901429, 2024-08-30T16:02:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:02:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:02:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:02:58.719 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:02:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:02:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:02:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:02:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:58.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:02:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:02:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:02:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:02:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:58.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:02:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:02:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:02:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:02:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:02:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:02:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:02:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:02:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:02:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:02:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 90.225905312999998, 2024-08-30T16:02:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:02:58.723 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:02:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:02:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:02:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:02:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:02:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:02:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:02:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:02:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:02:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:02:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:02:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:02:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:02:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:02:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:02:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:02:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:02:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:02:58.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:02:58.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:02:58.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:02:58.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:02:58.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:02:58.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:02:58.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:02:58.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:02:58.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:02:58.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:02:58.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:02:58.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:02:58.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:02:58.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:02:58.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:58.727 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:58.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:58.727 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:02:58.728 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:02:58.728 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:02:58.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:02:58.774 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:58.771+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:58.857 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:02:58.854+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:02:58.869 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:58.869 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:58.869 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:58.869 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:58.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:02:58.958 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:58.956+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:59.048 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:59.048 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:59.048 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:59.048 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:59.048 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:59.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:02:59.058 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:59.057+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:59.159 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:59.158+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:59.187 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:59.188 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:02:59.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:02:59.188 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:02:59.188 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:59.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:02:59.260 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:02:59.258+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:02:59.325 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:59.325 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:59.325 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:59.325 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:59.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:02:59.376 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:59.373+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:59.462 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:02:59.461+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:02:59.501 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:59.501 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:59.502 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:59.502 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:59.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:02:59.563 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:02:59.561+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:02:59.644 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:02:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:02:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:02:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:02:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 218.36940735900001, 2024-08-30T16:02:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:02:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:02:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:02:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:02:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:02:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:02:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:02:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:02:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:02:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:02:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:02:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:02:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:02:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:02:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:02:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:02:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:02:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 218.09647999800001, 2024-08-30T16:02:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:02:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:02:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:02:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:02:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:02:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:02:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:02:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:02:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:02:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:02:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:02:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:02:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:02:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:02:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:02:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:02:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:02:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:02:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:02:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:02:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:02:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:02:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:02:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:02:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:02:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:02:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:02:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:02:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:02:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:02:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:02:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:02:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:02:59.654 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:59.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:02:59.663 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:59.662+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:59.765 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:02:59.763+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:02:59.800 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:59.800 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:59.800 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:02:59.800 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:59.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:02:59.866 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:59.863+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:02:59.935 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:02:59.935 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:02:59.935 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:02:59.935 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:02:59.935 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:02:59.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:02:59.988 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:02:59.984+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:00.067 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:00.065+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:00.114 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:03:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 214.73665087200001, 2024-08-30T16:03:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:03:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:03:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:03:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:03:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:03:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:03:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:03:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:03:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:03:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:03:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:03:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:03:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:03:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:03:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:03:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:03:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:03:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.119 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:00.119 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:00.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:00.206 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:00.204+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:00.269 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:00.267+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:00.292 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:00.292 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:00.292 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:00.293 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:00.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:00.386 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:00.384+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:00.471 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:00.471 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:00.471 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:00.471 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:00.471 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:00.471 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:00.469+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:00.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:00.571 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:00.570+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:00.673 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:00.673 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:00.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:00.673 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:00.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.673 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:00.673 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:00.674 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 216.219939165, 2024-08-30T16:03:00.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:03:00.674 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:00.674 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:00.674 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:00.674 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:00.674 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:00.674 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:00.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.674 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:00.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:00.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:00.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:00.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:00.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:00.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:03:00.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:03:00.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:00.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:03:00.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:03:00.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:00.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:03:00.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:03:00.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:00.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:03:00.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:03:00.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:00.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:03:00.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:03:00.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:03:00.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:03:00.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:03:00.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:00.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:03:00.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:03:00.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:00.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:03:00.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:03:00.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:03:00.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:03:00.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:03:00.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:00.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:03:00.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:03:00.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:00.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:03:00.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:03:00.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:00.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:03:00.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:03:00.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:00.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:03:00.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:03:00.680 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.680 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:00.680 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.680 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:03:00.680 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:00.681 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 215.10407283000001, 2024-08-30T16:03:00.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:03:00.681 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:00.681 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:00.681 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:00.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:00.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:00.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:00.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:00.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:00.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:03:00.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:00.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:00.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:03:00.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:03:00.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:00.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:03:00.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:03:00.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:00.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:03:00.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:03:00.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:00.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:03:00.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:03:00.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:00.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:03:00.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:03:00.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:00.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:03:00.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:03:00.684 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.684 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:00.684 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.685 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:00.685 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:00.685 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 215.044481332, 2024-08-30T16:03:00.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:03:00.685 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:00.685 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:00.685 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:00.686 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:00.686 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:00.686 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:03:00.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.686 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:00.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:00.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:00.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:00.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:00.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:00.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:03:00.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:03:00.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:00.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:03:00.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:03:00.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:00.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:03:00.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:03:00.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:00.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:03:00.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:03:00.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:00.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:03:00.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:03:00.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:00.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:03:00.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:03:00.689 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.689 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:00.689 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:03:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 215.01778264500001, 2024-08-30T16:03:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:03:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:00.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:00.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:00.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:00.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:00.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:00.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:03:00.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:00.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:00.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:03:00.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:03:00.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:00.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:03:00.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:03:00.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:00.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:03:00.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:03:00.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:00.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:03:00.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:03:00.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:00.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:03:00.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:03:00.694 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.694 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:00.694 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.695 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:00.695 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:00.695 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 214.150166117, 2024-08-30T16:03:00.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:03:00.696 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:00.696 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:00.696 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:00.696 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:00.696 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:00.696 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:00.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.696 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:00.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:00.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:00.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:00.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:00.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:00.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:03:00.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:00.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:00.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:03:00.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:03:00.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:00.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:03:00.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:03:00.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:00.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:03:00.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:03:00.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:00.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:03:00.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:03:00.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:00.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:03:00.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:03:00.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:00.701 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:03:00.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:03:00.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:00.701 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:03:00.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:03:00.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:00.701 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:03:00.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:03:00.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:00.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:03:00.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:03:00.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:00.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:03:00.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:03:00.703 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.703 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:00.703 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.703 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:03:00.703 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:00.703 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 213.99091903799999, 2024-08-30T16:03:00.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:03:00.703 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:00.703 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:00.703 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:00.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:03:00.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:03:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:03:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:03:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:03:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:03:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:00.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:03:00.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:03:00.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:00.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:03:00.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:03:00.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:00.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:03:00.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:03:00.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:00.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:03:00.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:03:00.707 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.707 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:00.707 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.707 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:00.707 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:00.707 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 213.71474497200001, 2024-08-30T16:03:00.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:03:00.708 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:00.708 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:00.708 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:00.708 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:00.708 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:00.708 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:00.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.708 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:00.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.708 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:00.708 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:00.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:00.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:00.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:00.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:03:00.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:00.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:00.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:03:00.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:03:00.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:00.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:03:00.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:03:00.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:00.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:03:00.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:03:00.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:00.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:03:00.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:03:00.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:00.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:03:00.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:03:00.711 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.711 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:00.711 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.712 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:00.712 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:00.712 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 213.056453393, 2024-08-30T16:03:00.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:03:00.712 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:00.712 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:00.712 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:00.712 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:00.712 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:00.712 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:03:00.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.713 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:00.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:00.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:00.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:00.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:00.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:00.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:03:00.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:00.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:00.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:03:00.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:03:00.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:00.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:03:00.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:03:00.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.715 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:00.715 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:03:00.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:03:00.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.715 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:00.715 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:03:00.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:03:00.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.715 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:00.715 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:03:00.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:03:00.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:00.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:03:00.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:03:00.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:00.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:03:00.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:03:00.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:00.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:03:00.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:03:00.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:00.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:03:00.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:03:00.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:00.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:03:00.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:03:00.717 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.717 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:00.718 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.718 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:03:00.718 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:00.718 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 212.96054010899999, 2024-08-30T16:03:00.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:03:00.718 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:00.718 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:00.718 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:00.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:00.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:00.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:00.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:00.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:00.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:03:00.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:00.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:00.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:03:00.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:03:00.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:00.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:03:00.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:03:00.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:00.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:03:00.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:03:00.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:00.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:03:00.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:03:00.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:00.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:03:00.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:03:00.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:00.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:03:00.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:03:00.722 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.722 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:00.722 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.722 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:00.722 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:00.722 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 153.70917959799999, 2024-08-30T16:03:00.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:03:00.722 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:00.722 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:00.722 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:00.723 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:00.723 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:00.723 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:00.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.723 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:00.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:00.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:00.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:00.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:00.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:00.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:03:00.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:03:00.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:00.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:03:00.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:03:00.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:00.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:03:00.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:03:00.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:00.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:03:00.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:03:00.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:00.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:03:00.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:03:00.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:00.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:03:00.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:03:00.728 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.728 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:00.728 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.728 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:03:00.728 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:00.728 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 93.700074911000002, 2024-08-30T16:03:00.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:03:00.729 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:00.729 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:00.729 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:00.729 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:00.729 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:00.729 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:00.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.729 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:00.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:00.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:00.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:00.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:00.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:00.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:00.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:03:00.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:03:00.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:00.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:03:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:03:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:03:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:03:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:03:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:03:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:00.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:03:00.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:03:00.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:00.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:00.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:00.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:03:00.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:03:00.732 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.732 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:00.732 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.732 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:00.732 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:00.732 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:00.733 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:00.671+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:00.734 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:00.779 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:00.777+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:00.873 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:00.874 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:00.874 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:00.872+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:00.883 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:00.976 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:00.974+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:01.048 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:01.048 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:01.048 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:01.048 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:01.048 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:01.057 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:01.101 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:01.099+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:01.178 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:01.176+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:01.180 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:01.180 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:01.181 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:03:01.181 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:01.191 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:01.279 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:01.277+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:01.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-08-30T16:03:01.362 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:01.362 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:01.362 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:01.362 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:01.371 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:01.412 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:01.410+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:01.482 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:01.479+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:01.535 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:01.535 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:01.535 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:01.535 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:01.535 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:01.545 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:01.589 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:01.587+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:01.684 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:01.682+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:01.719 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:01.719 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:01.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:01.719 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:01.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.719 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:01.719 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:01.719 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 230.85592191699999, 2024-08-30T16:03:01.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:03:01.719 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.720 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.720 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.720 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.720 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.720 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:03:01.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.720 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:01.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:01.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:03:01.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:03:01.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:03:01.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:03:01.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:03:01.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:03:01.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:03:01.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:03:01.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:03:01.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:03:01.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:03:01.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:03:01.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:03:01.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.723 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.723 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:03:01.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:03:01.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:03:01.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:03:01.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:03:01.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:03:01.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:03:01.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:03:01.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:03:01.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:03:01.725 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.725 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.725 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.726 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:01.726 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:01.726 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 229.939505897, 2024-08-30T16:03:01.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:03:01.726 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.726 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.726 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.726 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.726 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.726 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:03:01.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.726 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:01.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:01.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:03:01.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:01.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:03:01.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:03:01.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:03:01.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:03:01.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:03:01.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:03:01.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:03:01.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:03:01.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.729 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.729 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:03:01.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:03:01.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:03:01.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:03:01.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:03:01.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:03:01.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:03:01.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:03:01.731 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.731 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.731 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.731 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:01.731 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:01.731 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 229.93780134400001, 2024-08-30T16:03:01.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:03:01.732 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.732 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.732 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.732 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.732 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.732 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:03:01.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.732 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:01.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:01.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:03:01.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:03:01.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:03:01.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:03:01.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.734 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.734 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:03:01.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:03:01.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:03:01.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:03:01.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:03:01.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:03:01.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:03:01.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:03:01.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:03:01.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:03:01.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:03:01.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:03:01.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:03:01.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:03:01.738 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.738 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.739 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.739 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.739 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:01.739 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 229.51834753700001, 2024-08-30T16:03:01.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:03:01.739 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.739 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.739 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.739 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:03:01.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.740 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:01.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:01.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:03:01.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:03:01.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:03:01.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:03:01.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:03:01.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:03:01.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:03:01.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:03:01.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:03:01.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:03:01.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:03:01.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:03:01.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:03:01.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:03:01.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:03:01.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:03:01.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:03:01.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:03:01.744 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.744 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.744 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.744 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.745 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:01.745 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 229.50145936300001, 2024-08-30T16:03:01.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:03:01.745 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.745 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.745 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.745 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.745 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.745 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:03:01.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.745 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.745 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:01.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:01.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:03:01.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:03:01.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:03:01.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:03:01.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:03:01.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:03:01.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:03:01.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:03:01.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:03:01.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:03:01.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:03:01.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:03:01.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:03:01.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:03:01.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:03:01.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:03:01.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:03:01.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:03:01.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.750 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.750 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.750 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:01.750 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 229.48845929399999, 2024-08-30T16:03:01.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:03:01.750 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.750 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.750 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.751 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:03:01.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.751 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:01.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:01.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:03:01.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:03:01.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:03:01.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:03:01.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:03:01.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:03:01.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:03:01.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:03:01.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:03:01.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:03:01.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:03:01.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:03:01.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:03:01.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:03:01.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:03:01.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:03:01.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:03:01.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:03:01.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.755 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.756 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.756 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:01.756 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 229.485967002, 2024-08-30T16:03:01.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:03:01.756 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.756 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.756 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.756 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.756 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.756 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:03:01.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.756 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:01.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:01.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:03:01.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:01.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:03:01.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:03:01.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:03:01.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:03:01.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:03:01.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:03:01.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:03:01.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:03:01.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:03:01.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:03:01.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:03:01.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:03:01.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:03:01.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:03:01.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:03:01.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:03:01.762 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.762 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.762 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.762 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.762 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:01.762 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 229.48520955399999, 2024-08-30T16:03:01.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:03:01.762 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.762 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.763 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.763 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.763 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.763 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:03:01.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.763 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:01.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:01.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:03:01.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:03:01.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:03:01.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:03:01.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:03:01.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:03:01.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:03:01.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:03:01.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:03:01.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:03:01.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:03:01.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:03:01.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:03:01.767 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.767 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.767 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.767 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.767 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 229.48447710400001, 2024-08-30T16:03:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:03:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:03:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:01.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:03:01.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:01.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:03:01.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:03:01.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:03:01.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:03:01.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:03:01.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:03:01.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:03:01.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:03:01.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:03:01.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:03:01.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:03:01.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:03:01.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:03:01.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:03:01.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:03:01.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:03:01.772 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.772 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.773 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.773 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.773 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:01.773 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 229.481938305, 2024-08-30T16:03:01.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:03:01.773 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.773 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.774 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.774 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.774 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.774 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:03:01.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.774 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:01.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:01.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:03:01.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:01.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:03:01.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:03:01.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:03:01.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:03:01.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:03:01.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:03:01.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:03:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:03:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:03:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:03:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:03:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:03:01.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:03:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:03:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:03:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:03:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.779 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.779 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:01.779 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 229.48119030399999, 2024-08-30T16:03:01.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:03:01.779 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.779 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.779 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.779 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.779 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.780 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:03:01.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.780 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:01.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:01.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:03:01.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:03:01.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:03:01.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:03:01.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:03:01.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:03:01.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:03:01.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:03:01.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:03:01.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:03:01.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:03:01.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:03:01.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:03:01.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:03:01.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:03:01.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:03:01.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:03:01.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:03:01.784 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.784 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.784 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.784 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.785 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:01.785 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 228.709073832, 2024-08-30T16:03:01.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:03:01.785 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.785 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.785 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.785 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.785 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.785 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:03:01.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.785 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:01.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:01.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:03:01.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:03:01.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:03:01.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:03:01.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:03:01.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:03:01.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:03:01.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:03:01.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:03:01.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:03:01.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:03:01.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:03:01.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:03:01.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:03:01.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:03:01.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:03:01.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:03:01.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:03:01.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.790 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.790 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.790 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.790 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:01.790 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 228.707016686, 2024-08-30T16:03:01.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:03:01.790 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.790 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.790 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.791 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.791 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.791 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:03:01.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.791 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:01.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:01.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:03:01.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:01.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:03:01.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:03:01.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:03:01.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:03:01.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:03:01.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:03:01.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:03:01.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:03:01.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:03:01.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:03:01.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:03:01.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:03:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:03:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:03:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:03:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:03:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.796 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.796 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:01.796 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 228.444072186, 2024-08-30T16:03:01.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:03:01.796 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.796 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.796 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.796 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.796 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.796 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:03:01.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.796 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:01.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:01.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:03:01.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:03:01.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:03:01.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:03:01.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:03:01.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:03:01.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:03:01.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:03:01.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:03:01.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:03:01.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:03:01.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:03:01.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:03:01.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:03:01.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:03:01.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:03:01.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:03:01.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:03:01.801 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.801 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.801 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.801 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.801 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:01.801 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 228.442980683, 2024-08-30T16:03:01.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:03:01.801 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.802 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.802 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.802 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.802 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.802 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:03:01.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.802 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:01.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:01.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:03:01.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:01.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:03:01.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:03:01.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:03:01.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:03:01.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:03:01.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:03:01.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:03:01.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:03:01.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:03:01.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:03:01.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:03:01.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:03:01.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:03:01.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:03:01.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:03:01.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:03:01.806 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.806 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.806 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.807 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.807 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:01.807 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 228.44223109800001, 2024-08-30T16:03:01.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:03:01.807 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.807 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.807 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.807 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.807 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.807 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:03:01.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.808 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:01.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:01.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:03:01.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:01.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:03:01.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:03:01.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:03:01.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:03:01.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:03:01.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:03:01.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:03:01.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:03:01.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:03:01.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:03:01.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:03:01.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:03:01.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:03:01.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:03:01.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:03:01.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:03:01.812 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.812 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.812 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.812 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.812 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:01.812 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 228.40849956599999, 2024-08-30T16:03:01.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:03:01.812 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.813 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.813 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.813 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.813 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.813 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:03:01.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.813 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:01.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:01.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:03:01.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:03:01.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:03:01.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:03:01.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:03:01.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:03:01.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:03:01.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:03:01.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:03:01.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:03:01.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:03:01.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:03:01.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:03:01.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:03:01.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:03:01.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:03:01.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:03:01.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:03:01.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.817 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.818 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.818 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:01.818 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 228.40797355800001, 2024-08-30T16:03:01.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:03:01.818 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.818 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.818 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.818 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.818 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.818 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:03:01.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.819 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:01.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:01.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:03:01.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:01.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:03:01.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:03:01.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.820 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.820 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:03:01.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:03:01.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:03:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:03:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:03:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:03:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:03:01.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:03:01.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:03:01.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:03:01.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:03:01.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:03:01.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:03:01.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:03:01.823 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.823 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.823 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.823 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:01.823 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:01.823 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 228.40751799500001, 2024-08-30T16:03:01.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:03:01.823 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.824 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.824 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.824 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.824 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.824 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:03:01.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.824 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:01.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:01.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:03:01.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:03:01.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:03:01.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:03:01.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:03:01.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:03:01.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:03:01.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:03:01.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:03:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:03:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:03:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:03:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:03:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:03:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:03:01.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:03:01.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:03:01.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:03:01.828 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.828 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.828 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.829 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:03:01.829 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:01.829 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 223.737002679, 2024-08-30T16:03:01.829 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:03:01.829 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:01.829 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:01.829 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:01.829 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:01.829 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:01.829 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:03:01.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.829 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:01.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:01.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:01.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:01.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:01.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:01.830 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.830 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.830 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:01.830 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:03:01.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:01.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:01.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:03:01.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:03:01.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:01.831 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:03:01.831 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:03:01.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:01.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:03:01.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:03:01.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:01.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:03:01.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:03:01.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:01.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:03:01.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:03:01.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:01.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:03:01.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:03:01.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:01.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:03:01.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:03:01.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:01.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:01.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:01.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:03:01.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:03:01.834 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.834 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.834 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.834 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:01.834 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:01.834 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:01.835 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:03:01.836 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:01.833+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:01.937 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:01.935+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:02.001 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:02.001 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:02.001 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:02.001 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:02.011 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:03:02.055 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:02.053+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:02.139 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:02.139 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:02.139 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:02.139 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:02.139 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:02.140 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:02.137+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:02.148 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:03:02.240 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:02.238+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:02.316 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:02.316 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:02.316 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:02.316 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:02.316 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.316 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:03:02.316 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:03:02.316 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 215.786522021, 2024-08-30T16:03:02.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:03:02.317 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:02.317 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:02.317 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:02.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:02.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:03:02.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:02.317 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.317 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.317 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:02.317 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:03:02.317 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:02.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:02.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:03:02.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:03:02.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:02.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:03:02.318 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:03:02.318 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.318 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.318 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:02.318 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:03:02.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:03:02.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:02.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:03:02.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:03:02.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:02.319 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:03:02.319 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:03:02.319 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.319 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.319 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:02.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:03:02.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:03:02.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:02.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:03:02.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:03:02.320 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.320 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.320 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:02.320 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:03:02.320 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:03:02.321 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:02.321 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:02.321 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:02.321 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.321 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.321 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:03:02.321 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:03:02.321 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 214.668073875, 2024-08-30T16:03:02.321 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:03:02.322 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:02.322 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:02.322 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:02.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:02.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:03:02.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:02.322 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.322 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.322 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:02.322 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:03:02.322 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:02.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:02.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:03:02.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:03:02.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:02.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:03:02.323 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:03:02.323 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.323 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.323 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:02.323 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:03:02.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:03:02.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:02.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:03:02.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:03:02.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.324 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:02.324 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:03:02.324 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:03:02.324 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.324 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:02.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:03:02.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:03:02.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:02.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:03:02.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:03:02.325 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.325 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.325 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:02.325 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:03:02.325 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:03:02.326 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:02.326 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:02.326 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:02.326 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:02.326 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:02.326 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:02.327 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:02.341 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:02.339+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:02.442 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:02.440+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:02.495 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:02.495 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:02.495 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:02.495 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:02.504 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:02.543 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:02.540+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:02.643 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:02.641+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:02.668 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:02.668 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:02.668 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:02.668 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:02.669 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:02.677 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:02.744 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:02.742+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:02.805 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:02.805 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:02.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 214.42575362900001, 2024-08-30T16:03:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:03:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:03:02.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.807 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:02.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:02.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:02.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:02.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:02.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:02.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:02.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:02.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:03:02.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:03:02.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:02.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:03:02.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:03:02.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:02.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:03:02.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:03:02.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:02.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:03:02.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:03:02.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:02.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:03:02.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:03:02.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:02.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:03:02.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:03:02.810 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:02.810 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:02.810 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:02.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.810 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:03:02.810 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:02.810 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 94.312757512999994, 2024-08-30T16:03:02.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:03:02.810 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:02.810 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:02.810 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:02.810 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:02.810 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:02.811 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:03:02.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.811 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:02.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:02.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:02.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:02.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:02.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:02.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:02.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:02.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:03:02.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:02.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:02.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:03:02.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:03:02.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:02.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:03:02.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:03:02.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:02.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:03:02.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:03:02.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:02.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:03:02.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:03:02.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:02.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:02.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:02.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:03:02.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:03:02.814 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:02.814 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:02.814 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:02.814 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:02.815 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:02.815 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:02.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:02.857 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:02.855+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:02.946 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:02.944+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:02.955 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:02.956 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:02.956 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:02.956 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:02.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:03.049 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:03.047+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:03.136 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:03.136 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:03.136 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:03.136 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:03.136 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:03.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:03.147 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:03.146+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:03.249 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:03.246+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:03.312 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:03.312 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:03.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:03.313 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:03:03.313 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:03.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:03.362 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:03.360+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:03.450 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:03.448+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:03.490 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:03.490 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:03.490 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:03.490 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:03.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:03.583 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:03.581+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:03.652 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:03.650+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:03.667 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:03.667 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:03.667 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:03.667 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:03.667 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:03.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:03.753 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:03.752+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:03.847 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:03.847 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:03.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 222.57301418200001, 2024-08-30T16:03:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:03:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:03:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:03:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:03:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:03:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:03:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:03:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:03:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:03:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:03:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:03:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:03:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:03:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:03:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:03:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 222.30008682100001, 2024-08-30T16:03:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:03:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:03:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:03:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:03:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:03:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:03:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:03:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:03:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:03:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:03:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:03:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:03:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:03:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:03:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:03:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:03.857 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:03.857 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:03.857 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:03.857 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:03.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:03.858 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:03.856+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:03.959 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:03.957+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:04.025 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:04.025 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:04.025 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:04.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:04.077 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:04.073+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:04.161 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:04.159+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:04.200 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:04.201 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:04.201 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:04.201 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:04.201 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:04.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:04.262 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:04.261+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:04.363 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:04.361+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:04.384 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:04.385 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:04.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:04.385 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:04.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:04.385 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:03:04.385 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:04.385 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 219.007291001, 2024-08-30T16:03:04.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:03:04.385 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:04.385 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:04.385 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:04.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:04.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:04.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:04.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:04.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:04.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:04.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:04.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:04.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:04.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:04.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:04.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:04.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:03:04.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:03:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:03:04.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:04.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:04.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:04.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:03:04.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:03:04.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:04.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:04.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:04.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:03:04.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:03:04.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:04.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:04.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:04.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:03:04.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:03:04.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:04.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:04.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:04.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:03:04.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:03:04.389 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:04.389 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:04.389 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:04.389 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:04.389 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:04.389 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:04.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:04.465 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:04.463+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:04.561 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:04.561 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:04.561 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:04.561 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:04.565 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:04.564+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:04.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:04.666 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:04.664+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:04.791 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:04.792 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:04.792 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:04.792 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:04.792 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:04.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:04.793 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:04.790+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:04.893 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:04.890+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:04.984 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T16:03:04.971+0000 12094640 -1 mon.a@0(leader).osd e63 definitely_dead 0 2024-08-30T16:03:04.994 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:04.991+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:05.025 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:05.025 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 220.56254945699999, 2024-08-30T16:03:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:03:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:05.026 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:05.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:03:05.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:03:05.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:05.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:03:05.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:03:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:03:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:03:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:03:05.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:03:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:03:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:03:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:03:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:03:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:03:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:05.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:03:05.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:03:05.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:05.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:03:05.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:03:05.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:03:05.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:03:05.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:03:05.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:05.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:03:05.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:03:05.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:03:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:03:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:03:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:03:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:03:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:03:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:03:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 219.446683122, 2024-08-30T16:03:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:03:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:03:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:03:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:03:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:03:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:03:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:03:05.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:03:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:03:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:03:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:03:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:03:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:03:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:03:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 219.38709162399999, 2024-08-30T16:03:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:03:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:05.038 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:03:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:05.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:05.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:03:05.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:03:05.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:05.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:03:05.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:03:05.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:05.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:03:05.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:03:05.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:05.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:03:05.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:03:05.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:05.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:03:05.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:03:05.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:05.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:03:05.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:03:05.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.043 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:05.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.043 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:03:05.043 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:05.043 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 219.360392937, 2024-08-30T16:03:05.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:03:05.043 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:05.043 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:05.043 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:05.043 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:05.044 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:05.044 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:05.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.044 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:05.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:05.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:05.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:05.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:05.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:05.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:03:05.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:05.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:05.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:03:05.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:03:05.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:05.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:03:05.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:03:05.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:05.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:03:05.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:03:05.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:05.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:03:05.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:03:05.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:05.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:03:05.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:03:05.047 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.047 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:05.047 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.047 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:05.047 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:05.047 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 218.49277640899999, 2024-08-30T16:03:05.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:03:05.047 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:05.047 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:05.048 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:05.048 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:05.048 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:05.048 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:05.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.048 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:05.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:05.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:05.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:03:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:03:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:03:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:05.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:03:05.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:03:05.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:05.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:03:05.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:03:05.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:05.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:03:05.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:03:05.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:05.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:03:05.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:03:05.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:05.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:03:05.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:03:05.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:05.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:03:05.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:03:05.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:05.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:03:05.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:03:05.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:05.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:03:05.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:03:05.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:05.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:03:05.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:03:05.053 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.053 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:05.053 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.054 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:03:05.054 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:05.054 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 218.33352933, 2024-08-30T16:03:05.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:03:05.054 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:05.054 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:05.054 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:05.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:05.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:05.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:05.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:05.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:05.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:03:05.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:03:05.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:05.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:03:05.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:03:05.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:05.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:03:05.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:03:05.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:05.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:03:05.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:03:05.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:05.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:03:05.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:03:05.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:05.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:03:05.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:03:05.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:05.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:03:05.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:03:05.058 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.058 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:05.058 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.059 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:05.059 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:05.059 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 218.05735526399999, 2024-08-30T16:03:05.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:03:05.059 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:05.059 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:05.059 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:05.059 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:05.059 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:05.059 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:05.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:05.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:03:05.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:05.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:05.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:03:05.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:03:05.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:05.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:03:05.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:03:05.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:05.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:03:05.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:03:05.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:05.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:03:05.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:03:05.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:05.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:03:05.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:03:05.063 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.063 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:05.063 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.063 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:05.063 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:05.063 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 217.39906368499999, 2024-08-30T16:03:05.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:03:05.064 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:05.064 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:05.064 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:05.064 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:05.064 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:05.064 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:03:05.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.064 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:05.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:05.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:05.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:03:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:03:05.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:03:05.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:05.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:03:05.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:03:05.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:05.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:03:05.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:03:05.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:05.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:03:05.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:03:05.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:05.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:03:05.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:03:05.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:05.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:03:05.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:03:05.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:05.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:03:05.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:03:05.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:05.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:03:05.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:03:05.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:05.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:03:05.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:03:05.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:05.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:03:05.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:03:05.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.069 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:05.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.069 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:03:05.069 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:05.069 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 217.30315040100001, 2024-08-30T16:03:05.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:03:05.070 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:05.070 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:05.070 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:05.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:05.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:05.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:05.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:05.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:05.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:03:05.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:05.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:05.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:03:05.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:03:05.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:05.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:03:05.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:03:05.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:05.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:03:05.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:03:05.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:05.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:03:05.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:03:05.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:05.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:03:05.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:03:05.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:05.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:03:05.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:03:05.074 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.074 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:05.074 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.075 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:05.075 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:05.075 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 158.05178989000001, 2024-08-30T16:03:05.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:03:05.075 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:05.075 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:05.075 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:05.075 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:05.075 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:05.075 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:05.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.075 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:05.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:05.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:05.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:05.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:05.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:05.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:03:05.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:03:05.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:05.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:03:05.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:03:05.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:05.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:03:05.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:03:05.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:05.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:03:05.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:03:05.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:05.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:03:05.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:03:05.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:05.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:03:05.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:03:05.079 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.079 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:05.079 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.079 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:03:05.079 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:05.079 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 98.042685203000005, 2024-08-30T16:03:05.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:03:05.079 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:05.079 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:05.079 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:05.079 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:05.080 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:05.080 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:05.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.080 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:05.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:05.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:05.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:05.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:05.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:05.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:05.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:03:05.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:03:05.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:05.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:03:05.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:03:05.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:05.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:03:05.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:03:05.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:05.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:03:05.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:03:05.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:05.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:03:05.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:03:05.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:05.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:05.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:05.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:03:05.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:03:05.083 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.083 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:05.083 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.083 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:05.083 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:05.083 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:05.084 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:05.125 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:05.123+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:05.195 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:05.193+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:05.230 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:05.230 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:05.230 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:05.242 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:05.296 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:05.294+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:05.385 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:05.385 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:05.385 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:05.385 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:05.385 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:05.395 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:05.397 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:05.395+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:05.498 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:05.495+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:05.521 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:05.521 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:05.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:05.522 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:03:05.522 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:05.530 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:05.599 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:05.597+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:05.641 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:05.641 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:05.654 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:05.654 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:05.654 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:05.654 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:05.663 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:05.706 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:05.703+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:05.748 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:05.749 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:05.803 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":46,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:03:05.814 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:05.812+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:05.817 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2024-08-30T16:03:05.817 INFO:tasks.mon_recovery:causing some monitor log activity 2024-08-30T16:03:05.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '1 of 30' 2024-08-30T16:03:05.827 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:05.827 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:05.827 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:05.827 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:05.827 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:05.837 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:05.846 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:05.846 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:05.902 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:05.900+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:05.967 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:05.967 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:06.003 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:06.002+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:06.050 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:06.050 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:06.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:06.050 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:06.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.050 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:06.050 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:06.050 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 235.187024022, 2024-08-30T16:03:06.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:03:06.051 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.051 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.051 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.051 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.051 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.051 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:03:06.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.051 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:06.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:06.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.052 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.052 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:03:06.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:03:06.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:03:06.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:03:06.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:03:06.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:03:06.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:03:06.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:03:06.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:03:06.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:03:06.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:03:06.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:03:06.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:03:06.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:03:06.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:03:06.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:03:06.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:03:06.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:03:06.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:03:06.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:03:06.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:03:06.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:03:06.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:03:06.056 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.056 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.056 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.056 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:06.057 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:06.057 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 234.27060800199999, 2024-08-30T16:03:06.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:03:06.057 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.057 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.057 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.057 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.057 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.057 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:03:06.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.057 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:06.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:06.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.058 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:03:06.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:06.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:03:06.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:03:06.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:03:06.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:03:06.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:03:06.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:03:06.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:03:06.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:03:06.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:03:06.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:03:06.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:03:06.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:03:06.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:03:06.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:03:06.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:03:06.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:03:06.062 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.062 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.062 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.062 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:06.062 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 234.26890344899999, 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:06.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:03:06.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:03:06.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:03:06.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:03:06.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:03:06.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:03:06.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:03:06.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:03:06.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:03:06.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:03:06.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:03:06.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:03:06.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:03:06.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:03:06.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:03:06.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:03:06.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:03:06.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:03:06.067 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.067 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.067 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.067 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.068 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:06.068 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 233.849449642, 2024-08-30T16:03:06.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:03:06.068 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.068 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.068 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.068 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.068 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.068 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:03:06.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.068 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:06.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:06.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.069 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.069 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:03:06.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:03:06.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:03:06.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:03:06.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:03:06.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:03:06.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:03:06.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:03:06.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:03:06.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:03:06.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:03:06.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:03:06.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:03:06.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:03:06.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:03:06.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:03:06.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:03:06.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:03:06.072 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.073 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.073 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.073 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.073 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:06.073 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 233.83256146799999, 2024-08-30T16:03:06.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:03:06.073 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.073 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.073 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.073 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.073 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.074 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:03:06.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.074 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:06.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:06.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:03:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:03:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:03:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:03:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:03:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:03:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:03:06.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:03:06.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:03:06.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:03:06.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:03:06.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 233.81956139900001, 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.078 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:03:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:03:06.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:03:06.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:03:06.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:03:06.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:03:06.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:03:06.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:03:06.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:03:06.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:03:06.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:03:06.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:03:06.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:03:06.083 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.083 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.083 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.083 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.083 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:06.083 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 233.81706910700001, 2024-08-30T16:03:06.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:03:06.083 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.083 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.083 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.084 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.084 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.084 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:03:06.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.084 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:06.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:06.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:03:06.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:06.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:03:06.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:03:06.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:03:06.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:03:06.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:03:06.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:03:06.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:03:06.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:03:06.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:03:06.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:03:06.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:03:06.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:03:06.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:03:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:03:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:03:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:03:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.088 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.089 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:06.089 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 233.81631165900001, 2024-08-30T16:03:06.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:03:06.089 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.089 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.089 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:03:06.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.089 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:06.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:06.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:03:06.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:03:06.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:03:06.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:03:06.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:03:06.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:03:06.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:03:06.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:03:06.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:03:06.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:03:06.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:03:06.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:03:06.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:03:06.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:03:06.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:03:06.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:03:06.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:03:06.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:03:06.093 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.094 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.094 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.094 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.094 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:06.094 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 233.81557920899999, 2024-08-30T16:03:06.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:03:06.094 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.094 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.095 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:03:06.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.095 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:06.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:06.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:03:06.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:06.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:03:06.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:03:06.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:03:06.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:03:06.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:03:06.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:03:06.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:03:06.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:03:06.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:03:06.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:03:06.099 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.099 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.099 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.099 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.099 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:06.099 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 233.81304041000001, 2024-08-30T16:03:06.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:03:06.099 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.099 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.099 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:03:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:06.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:03:06.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:06.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:03:06.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:03:06.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:03:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:03:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:03:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:03:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:03:06.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:03:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:03:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:03:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:03:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:03:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:03:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:03:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:03:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:03:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 233.81229240900001, 2024-08-30T16:03:06.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:03:06.105 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.105 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.105 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:03:06.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.105 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.105 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.105 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:06.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:06.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:03:06.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:03:06.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:03:06.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:03:06.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:03:06.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:03:06.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:03:06.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:03:06.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:03:06.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:03:06.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:03:06.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:03:06.110 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.110 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.110 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.110 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.110 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:06.110 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 233.04017593699999, 2024-08-30T16:03:06.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:03:06.110 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.110 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.110 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.111 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.111 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.111 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:03:06.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.111 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:06.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:06.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:03:06.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:03:06.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:03:06.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:03:06.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:03:06.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:03:06.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:03:06.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:03:06.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:03:06.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:03:06.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:03:06.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:03:06.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:03:06.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:03:06.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:03:06.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:03:06.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:03:06.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:03:06.115 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.115 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.115 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.116 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.116 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:06.116 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 233.03811879099999, 2024-08-30T16:03:06.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:03:06.116 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.116 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.116 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.116 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.116 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.116 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:03:06.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.116 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:06.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:06.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:03:06.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:06.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:03:06.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:03:06.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:03:06.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:03:06.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:03:06.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:03:06.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:03:06.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:03:06.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:03:06.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:03:06.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:03:06.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:03:06.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:03:06.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:03:06.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:03:06.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:03:06.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.121 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.121 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.121 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:06.121 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 232.77517429100001, 2024-08-30T16:03:06.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:03:06.121 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.121 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.122 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.122 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.122 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.122 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:03:06.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.122 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:06.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:06.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:03:06.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:03:06.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:03:06.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:03:06.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:03:06.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:03:06.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:03:06.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:03:06.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:03:06.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:03:06.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:03:06.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:03:06.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:03:06.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:03:06.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:03:06.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:03:06.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:03:06.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:03:06.127 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.127 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.127 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.128 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.128 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:06.128 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 232.77408278799999, 2024-08-30T16:03:06.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:03:06.128 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.128 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.128 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.128 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.128 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.128 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:03:06.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.128 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:06.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:06.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:03:06.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:06.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:03:06.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:03:06.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:03:06.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:03:06.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:03:06.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:03:06.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:03:06.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:03:06.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:03:06.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:03:06.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:03:06.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:03:06.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:03:06.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:03:06.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:03:06.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:03:06.133 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.133 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.133 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.133 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.133 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:06.133 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 232.77333320299999, 2024-08-30T16:03:06.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:03:06.133 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.133 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.133 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.134 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.134 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.134 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:03:06.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.134 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:06.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:06.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:03:06.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:06.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:03:06.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:03:06.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:03:06.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:03:06.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:03:06.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:03:06.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:03:06.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:03:06.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:03:06.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:03:06.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:03:06.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:03:06.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:03:06.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:03:06.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:03:06.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:03:06.138 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.139 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.139 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.139 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.139 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:06.139 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 232.739601671, 2024-08-30T16:03:06.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:03:06.139 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.139 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.139 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.139 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.139 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.139 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:03:06.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.140 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:06.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:06.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:03:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:03:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:03:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:03:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:03:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:03:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:03:06.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:03:06.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:03:06.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:03:06.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:03:06.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:03:06.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:03:06.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:03:06.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:03:06.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:03:06.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:03:06.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:03:06.144 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.144 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.144 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.144 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.144 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:06.144 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 232.73907566299999, 2024-08-30T16:03:06.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:03:06.144 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.144 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.144 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:03:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:06.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:03:06.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:06.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:03:06.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:03:06.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:03:06.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:03:06.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:03:06.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:03:06.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:03:06.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:03:06.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:03:06.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:03:06.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:03:06.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:03:06.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:03:06.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:03:06.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:03:06.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:03:06.149 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.149 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.149 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.149 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:06.149 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:06.149 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 232.73862009999999, 2024-08-30T16:03:06.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:03:06.149 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.149 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.150 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.150 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.150 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.150 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:03:06.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.150 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:06.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:06.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:03:06.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:03:06.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:03:06.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:03:06.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:03:06.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:03:06.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:03:06.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:03:06.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:03:06.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:03:06.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:03:06.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:03:06.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:03:06.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:03:06.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:03:06.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:03:06.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:03:06.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:03:06.154 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.154 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.154 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.154 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:03:06.154 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:06.154 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 228.06810478400001, 2024-08-30T16:03:06.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:03:06.155 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:06.155 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:06.155 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:06.155 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:06.155 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:06.155 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:03:06.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.155 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:06.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:06.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:06.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:06.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:06.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:06.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:06.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:03:06.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:06.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:06.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:03:06.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:03:06.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:06.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:03:06.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:03:06.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:06.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:03:06.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:03:06.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:06.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:03:06.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:03:06.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:06.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:03:06.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:03:06.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:06.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:03:06.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:03:06.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:06.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:03:06.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:03:06.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:06.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:06.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:06.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:03:06.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:03:06.159 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.159 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.159 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.160 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:06.160 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:06.160 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:06.160 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:06.161 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:06.161 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:06.162 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:06.162 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:03:06.163 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-08-30T16:03:06.163 INFO:teuthology.orchestra.run.smithi050.stderr:marked out osd.2. 2024-08-30T16:03:06.163 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:06.159+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:06.230 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:03:06.230 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:03:06.262 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:06.260+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:06.314 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:06.314 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:06.328 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:06.328 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:06.337 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:03:06.379 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:06.377+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:06.463 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:06.460+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:06.500 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:06.501 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:06.509 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:03:06.563 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:06.562+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:06.635 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:06.636 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:06.644 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:06.664 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:06.662+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:06.765 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:06.764+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:06.809 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:06.809 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:06.818 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:06.866 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:06.864+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:06.967 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:06.965+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:06.981 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:06.981 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:06.991 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:07.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '2 of 30' 2024-08-30T16:03:07.075 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:07.073+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:07.160 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:07.160 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:07.169 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:07.166+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:07.170 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:07.270 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:07.268+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:07.302 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:07.302 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:07.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:07.395 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:07.393+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:07.472 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:07.471+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:07.490 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:07.490 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:07.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:07.573 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:07.572+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:07.674 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:07.672+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:07.675 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:07.675 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:07.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:07.775 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:07.774+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:07.855 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:07.855 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:07.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:07.909 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:07.906+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:07.977 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:07.976+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:08.004 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:08.004 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:08.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:08.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '3 of 30' 2024-08-30T16:03:08.083 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:08.081+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:08.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-08-30T16:03:08.178 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:08.177+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:08.224 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:08.224 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:08.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:08.280 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:08.278+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:08.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:03:08.381 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:08.380+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:08.410 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:08.411 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:08.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:08.505 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:08.503+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:08.584 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:08.582+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:08.604 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:08.604 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:08.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:08.685 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:08.683+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:08.780 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:08.780 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:08.785 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:08.784+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:08.797 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:08.797 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:08.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:08.890 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:08.889+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:08.910 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:08.910 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:08.983 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:08.983 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:08.987 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:08.986+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:08.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:09.021 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:09.022 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:09.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '4 of 30' 2024-08-30T16:03:09.088 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:09.086+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:09.131 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:09.131 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:09.168 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:09.168 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:09.168 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:09.169 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:09.169 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:09.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:09.213 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:09.214 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:09.214 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:09.212+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:09.289 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:09.287+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:09.295 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:09.295 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:09.335 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:09.335 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:09.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:09.335 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:09.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.335 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:09.335 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:09.335 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 224.882332303, 2024-08-30T16:03:09.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:03:09.335 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:09.336 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:09.336 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:09.336 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:09.336 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:09.336 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:09.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.336 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:09.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:09.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:09.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:09.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:09.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:09.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:03:09.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:03:09.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:09.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:03:09.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:03:09.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:09.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:03:09.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:03:09.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:09.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:03:09.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:03:09.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:09.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:03:09.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:03:09.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:03:09.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:03:09.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:03:09.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:09.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:03:09.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:03:09.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:09.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:03:09.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:03:09.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:03:09.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:03:09.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:03:09.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:09.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:03:09.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:03:09.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:09.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:03:09.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:03:09.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:09.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:03:09.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:03:09.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:09.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:03:09.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:03:09.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.342 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:09.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.342 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:03:09.342 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:09.342 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 223.76646596800001, 2024-08-30T16:03:09.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:03:09.343 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:09.343 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:09.343 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:09.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:09.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:09.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:09.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:09.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:09.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:03:09.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:09.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:09.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:03:09.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:03:09.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:09.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:03:09.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:03:09.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:09.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:03:09.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:03:09.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:09.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:03:09.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:03:09.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:09.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:03:09.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:03:09.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:09.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:03:09.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:03:09.346 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.346 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:09.346 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.347 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:09.347 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:09.347 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 223.70687447, 2024-08-30T16:03:09.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:03:09.347 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:09.347 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:09.347 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:09.347 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:09.347 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:09.347 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:03:09.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.347 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:09.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:09.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:09.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:09.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:09.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:09.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:03:09.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:03:09.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:09.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:03:09.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:03:09.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:09.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:03:09.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:03:09.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:09.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:03:09.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:03:09.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:09.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:03:09.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:03:09.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:09.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:03:09.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:03:09.351 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.351 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:09.351 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.351 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:03:09.351 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:09.351 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 223.68017578300001, 2024-08-30T16:03:09.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:03:09.351 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:09.351 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:09.351 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:09.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:09.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:03:09.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:09.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:09.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:03:09.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:03:09.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:09.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:03:09.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:03:09.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:09.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:03:09.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:03:09.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:09.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:03:09.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:03:09.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:09.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:03:09.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:03:09.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.355 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:09.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.355 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:09.355 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:09.355 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 222.812559255, 2024-08-30T16:03:09.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:03:09.356 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:09.356 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:09.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:09.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:09.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:09.356 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:09.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.356 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:09.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:09.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:09.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:09.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:09.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:09.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:03:09.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:09.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:09.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:03:09.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:03:09.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:09.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:03:09.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:03:09.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:09.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:03:09.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:03:09.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:09.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:03:09.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:03:09.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:09.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:03:09.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:03:09.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:09.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:03:09.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:03:09.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:09.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:03:09.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:03:09.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:09.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:03:09.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:03:09.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:09.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:03:09.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:03:09.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:09.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:03:09.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:03:09.361 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.361 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:09.361 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.362 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:03:09.362 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:09.362 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 222.65331217599999, 2024-08-30T16:03:09.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:03:09.362 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:09.362 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:09.362 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:09.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:09.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:09.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:09.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:09.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:09.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:03:09.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:03:09.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:09.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:03:09.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:03:09.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:09.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:03:09.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:03:09.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:09.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:03:09.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:03:09.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:09.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:03:09.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:03:09.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:09.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:03:09.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:03:09.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:09.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:03:09.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:03:09.365 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.365 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:09.366 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.366 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:09.366 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:09.366 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 222.37713811, 2024-08-30T16:03:09.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:03:09.366 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:09.366 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:09.366 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:09.366 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:09.366 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:09.366 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:09.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.367 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:09.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:09.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:09.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:09.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:09.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:09.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:03:09.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:09.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:09.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:03:09.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:03:09.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:09.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:03:09.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:03:09.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:09.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:03:09.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:03:09.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:09.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:03:09.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:03:09.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:09.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:03:09.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:03:09.370 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.370 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:09.370 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.370 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:09.370 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:09.370 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 221.718846531, 2024-08-30T16:03:09.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:03:09.370 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:09.370 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:09.371 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:09.371 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:09.371 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:09.371 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:03:09.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.371 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:09.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:09.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:09.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:09.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:09.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:09.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:03:09.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:09.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:03:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:03:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:03:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:03:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:03:09.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:03:09.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:09.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:03:09.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:03:09.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:09.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:03:09.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:03:09.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:09.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:03:09.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:03:09.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:09.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:03:09.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:03:09.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:09.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:03:09.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:03:09.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:09.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:03:09.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:03:09.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:09.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:03:09.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:03:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:03:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 221.62293324699999, 2024-08-30T16:03:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:03:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:09.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:09.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:09.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:09.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:09.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:03:09.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:09.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:09.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:03:09.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:03:09.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:09.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:03:09.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:03:09.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:09.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:03:09.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:03:09.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:09.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:03:09.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:03:09.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:09.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:03:09.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:03:09.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:09.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:03:09.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:03:09.381 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.381 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:09.381 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.381 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:09.381 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:09.382 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 162.37157273599999, 2024-08-30T16:03:09.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:03:09.382 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:09.382 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:09.382 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:09.382 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:09.382 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:09.382 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:09.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.383 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:09.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:09.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:09.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:09.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:09.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:09.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:03:09.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:03:09.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:09.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:03:09.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:03:09.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:09.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:03:09.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:03:09.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:09.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:03:09.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:03:09.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:09.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:03:09.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:03:09.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:09.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:03:09.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:03:09.386 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.386 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:09.386 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.386 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:03:09.386 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:09.386 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 102.362468049, 2024-08-30T16:03:09.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:03:09.386 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:09.386 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:09.386 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:09.387 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:09.387 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:09.387 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:09.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.387 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:09.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:09.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:09.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:09.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:09.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:09.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:09.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:03:09.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:03:09.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:09.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:03:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:03:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:03:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:03:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:03:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:03:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:09.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:03:09.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:03:09.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:09.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:09.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:09.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:03:09.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:03:09.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.391 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:09.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:09.391 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:09.391 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:09.393 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:09.393 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:03:09.394 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:03:09.394 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:09.392+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:09.458 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:09.459 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:09.494 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:09.492+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:09.521 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:09.521 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:09.521 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:09.530 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:09.594 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:09.593+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:09.598 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-08-30T16:03:09.694 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:09.695 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:09.695 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:09.695 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:09.695 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:09.695 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:09.694+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:09.703 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:09.796 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:09.794+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:09.827 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:09.827 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:09.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:09.827 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:03:09.827 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:09.837 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:09.921 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:09.919+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:09.998 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:09.996+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:09.999 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:09.999 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:09.999 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:10.000 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:10.009 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:10.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '5 of 30' 2024-08-30T16:03:10.107 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:10.101+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:10.185 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:10.185 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:10.185 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:10.185 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:10.185 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:10.194 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:10.200 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:10.199+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:10.301 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:10.299+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:10.401 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:10.400+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:10.411 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:10.412 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:10.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:10.412 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:10.412 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.412 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:10.412 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:10.412 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 239.54813165900001, 2024-08-30T16:03:10.412 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:03:10.412 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.412 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.412 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.413 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.413 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:03:10.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.413 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:10.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.413 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.413 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.413 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.413 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:10.413 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:03:10.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:03:10.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.414 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.414 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:03:10.414 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:03:10.414 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.414 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:03:10.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:03:10.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:03:10.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:03:10.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.415 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.415 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.415 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:03:10.415 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:03:10.415 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:03:10.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:03:10.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:03:10.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.416 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.416 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:03:10.416 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:03:10.416 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.416 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:03:10.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:03:10.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.417 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.417 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.417 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:03:10.417 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:03:10.417 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:03:10.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:03:10.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.418 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.418 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.418 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:03:10.418 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:03:10.418 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.418 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.418 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.418 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.419 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.419 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:10.419 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:10.419 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 238.63171563899999, 2024-08-30T16:03:10.419 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:03:10.419 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.419 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.419 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.419 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.419 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.419 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:03:10.419 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.419 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:10.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:10.420 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.420 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.420 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.420 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.420 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:03:10.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:10.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:03:10.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:03:10.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.421 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.421 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:03:10.421 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:03:10.421 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.421 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:03:10.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:03:10.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.422 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:03:10.422 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:03:10.422 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.422 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.422 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:03:10.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:03:10.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.423 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.423 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:03:10.423 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:03:10.423 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.423 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:03:10.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:03:10.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.424 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.424 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.424 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:03:10.424 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:03:10.424 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.424 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.424 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.424 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.425 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.425 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:10.425 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:10.425 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 238.630011086, 2024-08-30T16:03:10.425 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:03:10.425 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.425 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.425 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.425 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.425 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.425 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:03:10.425 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.425 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:10.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:10.426 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.426 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.426 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.426 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.426 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:03:10.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:03:10.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:03:10.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:03:10.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.427 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.427 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:03:10.427 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:03:10.427 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.427 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:03:10.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:03:10.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:03:10.428 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:03:10.428 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.428 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.428 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.428 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:03:10.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:03:10.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:03:10.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:03:10.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.429 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.429 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:03:10.429 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:03:10.429 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.429 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.430 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.430 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:03:10.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:03:10.430 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.430 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.430 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.430 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.430 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.430 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.430 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:10.430 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 238.210557279, 2024-08-30T16:03:10.430 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:03:10.430 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.431 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.431 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.431 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.431 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.431 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:03:10.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.431 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.431 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.431 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:10.431 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.431 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.431 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:10.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:03:10.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:03:10.432 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.432 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.432 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.432 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:03:10.432 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:03:10.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:03:10.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:03:10.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.433 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:03:10.433 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:03:10.433 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.433 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.433 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:03:10.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:03:10.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:03:10.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:03:10.434 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.434 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.434 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.434 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:03:10.434 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:03:10.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:03:10.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:03:10.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.435 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.435 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.435 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:03:10.435 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:03:10.435 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.435 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.435 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.435 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.436 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.436 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.436 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:10.436 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 238.193669105, 2024-08-30T16:03:10.436 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:03:10.436 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.436 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.436 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.436 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.436 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.436 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:03:10.436 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.437 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:10.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.437 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.437 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:10.437 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.437 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.437 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:03:10.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:03:10.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:03:10.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:03:10.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.438 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.438 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.438 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:03:10.438 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:03:10.438 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:03:10.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:03:10.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.439 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:03:10.439 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:03:10.439 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.439 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.439 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:03:10.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:03:10.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:03:10.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:03:10.440 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.440 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.440 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.440 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:03:10.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:03:10.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.441 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.441 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:03:10.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:03:10.441 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.441 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.441 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.441 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.441 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.441 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:10.441 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 238.18066903600001, 2024-08-30T16:03:10.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:03:10.442 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.442 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.442 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:03:10.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.442 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.442 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.442 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:10.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:10.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:03:10.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:03:10.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:03:10.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:03:10.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:03:10.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:03:10.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:03:10.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:03:10.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:03:10.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:03:10.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:03:10.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:03:10.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:03:10.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:03:10.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:03:10.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:03:10.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:03:10.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:03:10.447 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.447 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.447 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.447 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.447 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:10.447 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 238.17817674400001, 2024-08-30T16:03:10.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:03:10.447 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.448 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.448 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.448 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.448 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.448 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:03:10.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.448 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:10.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:10.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:03:10.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:10.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:03:10.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:03:10.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:03:10.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:03:10.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:03:10.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:03:10.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:03:10.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:03:10.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:03:10.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:03:10.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:03:10.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:03:10.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:03:10.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:03:10.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:03:10.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:03:10.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.457 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.457 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.457 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.457 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:10.457 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 238.17741929600001, 2024-08-30T16:03:10.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:03:10.458 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.458 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.458 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.458 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:03:10.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.458 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:10.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:10.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:03:10.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:03:10.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:03:10.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:03:10.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:03:10.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:03:10.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:03:10.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:03:10.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:03:10.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:03:10.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:03:10.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:03:10.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:03:10.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:03:10.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:03:10.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:03:10.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:03:10.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:03:10.463 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.463 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.463 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.463 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.463 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:10.463 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 238.176686846, 2024-08-30T16:03:10.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:03:10.464 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.464 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.464 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.464 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.464 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.464 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:03:10.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.464 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:10.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:10.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:03:10.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:10.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:03:10.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:03:10.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:03:10.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:03:10.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.466 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:03:10.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:03:10.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:03:10.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:03:10.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:03:10.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:03:10.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:03:10.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:03:10.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:03:10.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:03:10.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:03:10.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:03:10.468 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.469 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.469 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.469 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.469 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:10.469 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 238.17414804699999, 2024-08-30T16:03:10.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:03:10.469 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.469 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.469 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.469 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.469 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.470 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:03:10.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.470 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:10.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:10.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:03:10.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:10.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:03:10.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:03:10.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:03:10.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:03:10.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:03:10.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:03:10.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:03:10.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:03:10.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:03:10.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:03:10.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:03:10.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:03:10.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:03:10.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:03:10.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:03:10.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:03:10.474 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.475 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.475 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.475 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.475 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:10.475 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 238.17340004600001, 2024-08-30T16:03:10.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:03:10.475 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.475 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.475 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.476 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:03:10.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.476 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:10.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:10.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:03:10.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:03:10.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:03:10.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:03:10.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:03:10.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:03:10.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:03:10.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:03:10.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:03:10.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:03:10.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:03:10.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:03:10.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:03:10.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:03:10.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:03:10.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:03:10.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:03:10.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:03:10.480 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.480 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.481 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.481 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.481 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:10.481 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 237.40128357399999, 2024-08-30T16:03:10.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:03:10.481 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.481 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.481 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.481 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:03:10.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.482 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:10.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.482 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:10.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:03:10.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:03:10.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:03:10.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:03:10.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:03:10.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:03:10.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:03:10.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:03:10.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:03:10.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:03:10.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:03:10.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:03:10.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:03:10.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:03:10.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:03:10.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:03:10.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:03:10.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:03:10.486 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.486 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.486 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.486 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.486 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:10.487 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 237.39922642799999, 2024-08-30T16:03:10.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:03:10.487 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.487 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.487 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.487 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.487 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.487 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:03:10.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.487 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:10.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:10.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:03:10.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:10.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:03:10.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:03:10.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:03:10.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:03:10.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:03:10.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:03:10.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:03:10.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:03:10.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:03:10.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:03:10.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:03:10.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:03:10.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:03:10.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:03:10.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:03:10.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:03:10.492 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.492 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.492 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.492 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.493 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:10.493 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 237.13628192799999, 2024-08-30T16:03:10.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:03:10.493 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.493 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.493 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.493 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.493 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.493 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:03:10.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.493 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:10.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:10.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:03:10.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:03:10.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:03:10.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:03:10.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:03:10.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:03:10.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:03:10.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:03:10.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:03:10.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:03:10.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:03:10.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:03:10.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:03:10.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:03:10.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:03:10.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:03:10.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:03:10.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:03:10.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.499 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.499 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.500 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.500 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:10.500 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 237.13519042499999, 2024-08-30T16:03:10.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:03:10.501 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.501 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.501 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.501 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.501 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.501 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:03:10.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.501 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:10.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:10.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:03:10.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:10.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:03:10.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:03:10.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:03:10.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:03:10.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:03:10.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:03:10.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:03:10.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:03:10.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:03:10.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:03:10.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:03:10.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:03:10.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:03:10.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:03:10.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:03:10.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:03:10.506 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.506 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.506 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.507 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.507 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:10.507 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 237.13444084, 2024-08-30T16:03:10.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:03:10.507 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.507 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.507 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.507 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.507 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.507 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:03:10.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.507 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:10.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:10.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:03:10.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:10.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:03:10.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:03:10.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:03:10.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:03:10.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:03:10.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:03:10.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:03:10.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:03:10.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:03:10.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:03:10.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:03:10.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:03:10.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:03:10.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:03:10.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:03:10.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:03:10.512 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.512 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.512 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.512 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.512 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:10.512 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 237.10070930800001, 2024-08-30T16:03:10.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:03:10.513 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.513 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.513 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.513 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.513 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.513 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:03:10.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.513 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:10.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:10.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:03:10.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:03:10.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:03:10.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:03:10.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:03:10.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:03:10.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:03:10.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:03:10.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:03:10.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:03:10.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:03:10.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:03:10.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:03:10.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:03:10.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:03:10.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:03:10.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:03:10.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:03:10.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.518 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.518 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.518 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:10.518 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 237.1001833, 2024-08-30T16:03:10.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:03:10.518 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.519 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.519 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:03:10.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.519 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:10.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:10.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:03:10.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:10.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:03:10.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:03:10.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:03:10.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:03:10.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:03:10.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:03:10.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:03:10.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:03:10.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:03:10.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:03:10.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:03:10.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:03:10.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:03:10.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:03:10.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:03:10.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:03:10.524 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.524 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.524 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.524 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:10.524 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:10.524 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 237.09972773699999, 2024-08-30T16:03:10.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:03:10.525 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.525 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.525 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.525 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.525 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.525 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:03:10.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.525 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:10.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:10.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:03:10.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:03:10.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:03:10.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:03:10.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:03:10.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:03:10.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:03:10.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:03:10.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:03:10.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:03:10.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:03:10.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:03:10.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:03:10.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:03:10.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:03:10.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:03:10.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:03:10.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:03:10.530 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.530 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.530 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.530 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:03:10.530 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:10.530 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 232.42921242099999, 2024-08-30T16:03:10.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:03:10.530 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:10.530 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:10.530 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:10.531 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:10.531 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:10.531 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:03:10.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.531 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:10.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:10.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:10.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:10.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:10.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:10.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:10.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:03:10.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:10.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:10.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:03:10.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:03:10.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:10.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:03:10.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:03:10.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:10.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:03:10.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:03:10.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:10.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:03:10.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:03:10.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:10.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:03:10.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:03:10.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:10.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:03:10.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:03:10.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:10.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:03:10.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:03:10.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:10.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:10.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:10.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:03:10.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:03:10.535 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.536 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.536 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:10.536 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:10.536 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:10.538 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:03:10.538 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:10.536+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:10.638 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:10.637+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:10.702 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:10.702 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:10.702 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:10.702 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:10.711 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:03:10.739 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:10.738+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:10.835 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:10.835 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:10.835 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:10.835 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:10.835 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:10.841 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:10.839+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:10.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:03:10.942 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:10.940+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:11.017 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:11.017 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:11.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:11.017 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:11.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.017 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:03:11.017 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:03:11.017 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 224.48749589900001, 2024-08-30T16:03:11.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:03:11.018 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:11.018 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:11.018 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:11.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:11.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:03:11.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:11.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:11.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:03:11.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:11.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:11.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:03:11.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:03:11.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:11.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:03:11.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:03:11.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:11.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:03:11.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:03:11.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:11.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:03:11.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:03:11.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:11.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:03:11.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:03:11.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:11.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:03:11.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:03:11.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:11.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:03:11.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:03:11.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:11.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:03:11.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:03:11.022 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:11.022 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:11.022 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:11.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.022 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:03:11.022 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:03:11.022 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 223.36904775299999, 2024-08-30T16:03:11.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:03:11.023 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:11.023 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:11.023 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:11.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:11.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:03:11.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:11.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:11.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:03:11.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:11.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:11.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:03:11.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:03:11.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:11.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:03:11.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:03:11.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:11.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:03:11.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:03:11.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:11.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:03:11.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:03:11.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:11.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:03:11.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:03:11.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:11.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:03:11.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:03:11.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:11.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:03:11.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:03:11.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:11.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:03:11.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:03:11.027 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:11.027 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:11.027 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:11.027 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:11.027 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:11.028 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:11.029 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:11.042 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:11.041+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:11.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '6 of 30' 2024-08-30T16:03:11.143 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:11.142+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:11.175 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7] out_osds: [6, 4, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:03:11.176 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:03:11.176 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-30T16:03:11.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-08-30T16:03:11.193 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:11.193 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:11.194 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:11.202 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:11.244 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:11.243+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:11.345 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:11.343+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:11.366 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:11.366 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:11.367 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:11.375 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:11.460 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:11.458+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:11.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:03:11.543 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:11.544 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:11.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:11.544 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:11.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.544 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:11.544 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:11.544 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 223.164049498, 2024-08-30T16:03:11.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:03:11.544 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:11.544 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:11.544 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:11.544 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:11.544 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:11.545 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:03:11.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.545 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:11.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:11.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:11.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:11.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:11.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:11.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:11.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:11.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:03:11.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:03:11.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:11.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:03:11.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:03:11.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:11.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:03:11.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:03:11.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:11.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:03:11.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:03:11.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:11.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:03:11.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:03:11.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:11.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:03:11.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:03:11.548 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:11.548 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:11.548 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:11.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.548 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:03:11.548 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:11.548 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 103.05105338200001, 2024-08-30T16:03:11.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:03:11.548 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:11.548 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:11.548 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:11.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:11.549 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:11.549 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:03:11.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.549 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:11.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:11.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:11.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:11.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:11.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:11.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:11.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:11.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:03:11.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:11.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:11.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:03:11.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:03:11.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:11.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:03:11.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:03:11.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:11.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:03:11.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:03:11.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:11.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:03:11.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:03:11.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:11.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:11.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:11.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:03:11.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:03:11.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:11.556 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:11.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:11.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:11.556 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:11.556 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:11.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:11.558 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:11.556+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:11.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-08-30T16:03:11.659 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:11.658+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:11.760 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:11.759+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:11.770 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:11.770 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:11.779 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:11.779 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:11.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:11.877 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:11.875+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:11.877 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:11.877 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:11.961 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:11.962 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:11.962 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:11.960+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:11.967 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:11.967 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:11.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:12.059 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:12.059 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:12.063 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:12.061+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:12.140 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:12.140 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:12.140 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:12.140 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:12.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:12.163 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:12.161+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:12.214 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:03:12.214 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:12.214 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 66, 2024-08-30T16:03:12.214 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "9d72e2e2-0554-42af-8e62-9b1a3a804112", 2024-08-30T16:03:12.215 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-08-30T15:58:20.129410+0000", 2024-08-30T16:03:12.215 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-08-30T16:03:09.195776+0000", 2024-08-30T16:03:12.215 INFO:teuthology.orchestra.run.smithi050.stdout: "last_up_change": "2024-08-30T15:59:19.280372+0000", 2024-08-30T16:03:12.215 INFO:teuthology.orchestra.run.smithi050.stdout: "last_in_change": "2024-08-30T16:03:05.847464+0000", 2024-08-30T16:03:12.215 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T16:03:12.215 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_num": 5799936, 2024-08-30T16:03:12.215 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_set": [ 2024-08-30T16:03:12.215 INFO:teuthology.orchestra.run.smithi050.stdout: "pglog_hardlimit", 2024-08-30T16:03:12.215 INFO:teuthology.orchestra.run.smithi050.stdout: "purged_snapdirs", 2024-08-30T16:03:12.215 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_deletes", 2024-08-30T16:03:12.215 INFO:teuthology.orchestra.run.smithi050.stdout: "sortbitwise" 2024-08-30T16:03:12.215 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:03:12.215 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_version": 7, 2024-08-30T16:03:12.216 INFO:teuthology.orchestra.run.smithi050.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T16:03:12.216 INFO:teuthology.orchestra.run.smithi050.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T16:03:12.216 INFO:teuthology.orchestra.run.smithi050.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T16:03:12.216 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_snapshot": "", 2024-08-30T16:03:12.216 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_max": 2, 2024-08-30T16:03:12.216 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd": 8, 2024-08-30T16:03:12.216 INFO:teuthology.orchestra.run.smithi050.stdout: "require_min_compat_client": "luminous", 2024-08-30T16:03:12.216 INFO:teuthology.orchestra.run.smithi050.stdout: "min_compat_client": "jewel", 2024-08-30T16:03:12.216 INFO:teuthology.orchestra.run.smithi050.stdout: "require_osd_release": "reef", 2024-08-30T16:03:12.216 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_crimson": false, 2024-08-30T16:03:12.216 INFO:teuthology.orchestra.run.smithi050.stdout: "pools": [ 2024-08-30T16:03:12.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.216 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 1, 2024-08-30T16:03:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".mgr", 2024-08-30T16:03:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-08-30T15:59:08.355540+0000", 2024-08-30T16:03:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-08-30T16:03:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-08-30T16:03:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-08-30T16:03:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 2, 2024-08-30T16:03:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 1, 2024-08-30T16:03:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-08-30T16:03:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-08-30T16:03:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-08-30T16:03:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T16:03:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T16:03:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-08-30T16:03:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-08-30T16:03:12.218 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 1, 2024-08-30T16:03:12.218 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 1, 2024-08-30T16:03:12.218 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 1, 2024-08-30T16:03:12.218 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 1, 2024-08-30T16:03:12.218 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 1, 2024-08-30T16:03:12.218 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-08-30T16:03:12.218 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-08-30T16:03:12.218 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-08-30T16:03:12.218 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-08-30T16:03:12.218 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-08-30T16:03:12.218 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-08-30T16:03:12.218 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-08-30T16:03:12.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.219 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "19", 2024-08-30T16:03:12.219 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-08-30T16:03:12.219 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T16:03:12.219 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T16:03:12.219 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-08-30T16:03:12.219 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-08-30T16:03:12.219 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-08-30T16:03:12.219 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-08-30T16:03:12.219 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-08-30T16:03:12.219 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-08-30T16:03:12.219 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-08-30T16:03:12.219 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-08-30T16:03:12.219 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-08-30T16:03:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-08-30T16:03:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-08-30T16:03:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-08-30T16:03:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-08-30T16:03:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-08-30T16:03:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-08-30T16:03:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T16:03:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T16:03:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T16:03:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-08-30T16:03:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-08-30T16:03:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-08-30T16:03:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-08-30T16:03:12.220 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-08-30T16:03:12.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.221 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-08-30T16:03:12.221 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-08-30T16:03:12.221 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-08-30T16:03:12.221 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-08-30T16:03:12.221 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-08-30T16:03:12.221 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T16:03:12.221 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-08-30T16:03:12.221 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-08-30T16:03:12.221 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-08-30T16:03:12.221 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-08-30T16:03:12.221 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-08-30T16:03:12.221 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-08-30T16:03:12.222 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_max": 32, 2024-08-30T16:03:12.222 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_min": 1 2024-08-30T16:03:12.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.222 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-08-30T16:03:12.222 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": {} 2024-08-30T16:03:12.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.222 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-08-30T16:03:12.222 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 8, 2024-08-30T16:03:12.222 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 8, 2024-08-30T16:03:12.222 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.25, 2024-08-30T16:03:12.222 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-08-30T16:03:12.222 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-08-30T16:03:12.222 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-08-30T16:03:12.223 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-08-30T16:03:12.223 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-08-30T16:03:12.223 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.223 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-08-30T16:03:12.223 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "rbd", 2024-08-30T16:03:12.223 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-08-30T15:59:21.696796+0000", 2024-08-30T16:03:12.223 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-08-30T16:03:12.223 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-08-30T16:03:12.223 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-08-30T16:03:12.223 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 2, 2024-08-30T16:03:12.223 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 1, 2024-08-30T16:03:12.223 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-08-30T16:03:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-08-30T16:03:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-08-30T16:03:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T16:03:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T16:03:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-08-30T16:03:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-08-30T16:03:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 8, 2024-08-30T16:03:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 8, 2024-08-30T16:03:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 8, 2024-08-30T16:03:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 8, 2024-08-30T16:03:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 8, 2024-08-30T16:03:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-08-30T16:03:12.224 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-08-30T16:03:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-08-30T16:03:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-08-30T16:03:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-08-30T16:03:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-08-30T16:03:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-08-30T16:03:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "28", 2024-08-30T16:03:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-08-30T16:03:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T16:03:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T16:03:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-08-30T16:03:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-08-30T16:03:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 3, 2024-08-30T16:03:12.225 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 28, 2024-08-30T16:03:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-08-30T16:03:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-08-30T16:03:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-08-30T16:03:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-08-30T16:03:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-08-30T16:03:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-08-30T16:03:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-08-30T16:03:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-08-30T16:03:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-08-30T16:03:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-08-30T16:03:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-08-30T16:03:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T16:03:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T16:03:12.226 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T16:03:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-08-30T16:03:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-08-30T16:03:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-08-30T16:03:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-08-30T16:03:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-08-30T16:03:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-08-30T16:03:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-08-30T16:03:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-08-30T16:03:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-08-30T16:03:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-08-30T16:03:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T16:03:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-08-30T16:03:12.227 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-08-30T16:03:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-08-30T16:03:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-08-30T16:03:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-08-30T16:03:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-08-30T16:03:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-08-30T16:03:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd": {} 2024-08-30T16:03:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-08-30T16:03:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 4.9699997901916504, 2024-08-30T16:03:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 4.9699997901916504, 2024-08-30T16:03:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.62999999523162842, 2024-08-30T16:03:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 3.130000114440918, 2024-08-30T16:03:12.228 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 3.130000114440918, 2024-08-30T16:03:12.229 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-08-30T16:03:12.229 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-08-30T16:03:12.229 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-08-30T16:03:12.229 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.229 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.229 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:03:12.229 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-08-30T16:03:12.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.229 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-08-30T16:03:12.229 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4", 2024-08-30T16:03:12.229 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:12.229 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:12.230 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:12.230 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:12.230 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:12.230 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:12.230 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 15, 2024-08-30T16:03:12.230 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 60, 2024-08-30T16:03:12.230 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6812", 2024-08-30T16:03:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:03:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.231 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.232 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6813", 2024-08-30T16:03:12.232 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:03:12.232 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.232 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.232 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:12.232 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.232 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.232 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6815", 2024-08-30T16:03:12.232 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:03:12.232 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.232 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.233 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:12.233 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.233 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.233 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6814", 2024-08-30T16:03:12.233 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:03:12.233 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.233 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.233 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6812/1957399494", 2024-08-30T16:03:12.233 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6813/1957399494", 2024-08-30T16:03:12.233 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6815/1957399494", 2024-08-30T16:03:12.234 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6814/1957399494", 2024-08-30T16:03:12.234 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:12.234 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:12.234 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:12.234 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.234 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-08-30T16:03:12.234 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "0fe11a1c-4326-4bfd-a2a4-bff14023f301", 2024-08-30T16:03:12.234 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:12.234 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:12.234 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:12.234 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 22, 2024-08-30T16:03:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 22, 2024-08-30T16:03:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6808", 2024-08-30T16:03:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:03:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.235 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.236 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:12.236 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.236 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.236 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6809", 2024-08-30T16:03:12.236 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:03:12.236 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.236 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.236 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:12.236 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.237 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.237 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6811", 2024-08-30T16:03:12.237 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:03:12.237 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.237 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.237 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:12.237 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.237 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.237 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6810", 2024-08-30T16:03:12.237 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:03:12.237 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.237 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.238 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6808/3208178756", 2024-08-30T16:03:12.238 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6809/3208178756", 2024-08-30T16:03:12.238 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6811/3208178756", 2024-08-30T16:03:12.238 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6810/3208178756", 2024-08-30T16:03:12.238 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:12.238 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:12.238 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:12.238 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.238 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-08-30T16:03:12.238 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "a9d5354b-3600-41ed-80c5-a3439da7aca6", 2024-08-30T16:03:12.239 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:12.239 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:03:12.239 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:03:12.239 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:12.239 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:12.239 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:12.239 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 16, 2024-08-30T16:03:12.239 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:03:12.239 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:12.239 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:12.239 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:12.239 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.240 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.240 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6804", 2024-08-30T16:03:12.240 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:03:12.240 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.240 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.240 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:12.240 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.240 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.240 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6805", 2024-08-30T16:03:12.240 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:03:12.240 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6807", 2024-08-30T16:03:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:03:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.242 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.242 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6806", 2024-08-30T16:03:12.242 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:03:12.242 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.242 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.242 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6804/287018800", 2024-08-30T16:03:12.242 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6805/287018800", 2024-08-30T16:03:12.242 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6807/287018800", 2024-08-30T16:03:12.242 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6806/287018800", 2024-08-30T16:03:12.242 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:12.242 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-08-30T16:03:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "78bcc398-a7e4-4efc-9836-1dfb94407c9c", 2024-08-30T16:03:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:03:12.243 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 64, 2024-08-30T16:03:12.244 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:12.244 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:12.244 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:12.244 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.244 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.244 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6800", 2024-08-30T16:03:12.244 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:03:12.244 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.244 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.244 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:12.244 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.245 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.245 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6801", 2024-08-30T16:03:12.245 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:03:12.245 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.245 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.245 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:12.245 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.245 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.245 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6803", 2024-08-30T16:03:12.245 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:03:12.245 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.246 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.246 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:12.246 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.246 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.246 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6802", 2024-08-30T16:03:12.246 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:03:12.246 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.246 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.246 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6800/1140717714", 2024-08-30T16:03:12.246 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6801/1140717714", 2024-08-30T16:03:12.246 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6803/1140717714", 2024-08-30T16:03:12.247 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6802/1140717714", 2024-08-30T16:03:12.247 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:12.247 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:12.247 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:12.247 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.247 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-08-30T16:03:12.247 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "53a1060b-7985-4c2c-820e-64b158056113", 2024-08-30T16:03:12.247 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:12.247 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:03:12.247 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:03:12.247 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 20, 2024-08-30T16:03:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 24, 2024-08-30T16:03:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6805", 2024-08-30T16:03:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:03:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.248 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6806", 2024-08-30T16:03:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:03:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.250 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6808", 2024-08-30T16:03:12.250 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:03:12.250 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.250 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.250 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:12.250 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.250 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.250 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6807", 2024-08-30T16:03:12.250 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:03:12.250 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.250 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.251 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6805/1468073345", 2024-08-30T16:03:12.251 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6806/1468073345", 2024-08-30T16:03:12.251 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6808/1468073345", 2024-08-30T16:03:12.251 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6807/1468073345", 2024-08-30T16:03:12.251 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:12.251 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:12.251 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:12.251 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.251 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-08-30T16:03:12.251 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "853a03f7-227d-4f98-9e5c-c29896820746", 2024-08-30T16:03:12.251 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 13, 2024-08-30T16:03:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 64, 2024-08-30T16:03:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.253 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6809", 2024-08-30T16:03:12.253 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:03:12.253 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.253 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.253 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:12.253 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.253 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.253 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6810", 2024-08-30T16:03:12.253 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:03:12.253 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.253 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.254 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:12.254 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.254 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.254 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6812", 2024-08-30T16:03:12.254 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:03:12.254 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.254 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.254 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:12.254 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.254 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.255 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6811", 2024-08-30T16:03:12.255 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:03:12.255 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.255 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.255 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6809/3876025616", 2024-08-30T16:03:12.255 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6810/3876025616", 2024-08-30T16:03:12.255 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6812/3876025616", 2024-08-30T16:03:12.255 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6811/3876025616", 2024-08-30T16:03:12.255 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:12.255 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:12.255 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:12.255 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.256 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-08-30T16:03:12.256 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "bc978060-bf28-4027-8bba-7d6d8659bfc5", 2024-08-30T16:03:12.256 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:12.256 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:03:12.256 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:03:12.256 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:12.256 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:12.256 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:12.256 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:03:12.256 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:03:12.256 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:12.256 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:12.257 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:12.257 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.257 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.257 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6801", 2024-08-30T16:03:12.257 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:03:12.257 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.257 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.257 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:12.257 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.257 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.258 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6802", 2024-08-30T16:03:12.258 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:03:12.258 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.258 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.258 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:12.258 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.258 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.258 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6804", 2024-08-30T16:03:12.258 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:03:12.258 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.259 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.259 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:12.259 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.259 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.259 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6803", 2024-08-30T16:03:12.259 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:03:12.259 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.259 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.259 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6801/128519122", 2024-08-30T16:03:12.259 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6802/128519122", 2024-08-30T16:03:12.260 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6804/128519122", 2024-08-30T16:03:12.260 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6803/128519122", 2024-08-30T16:03:12.260 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:12.260 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:12.260 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:12.260 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.260 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-08-30T16:03:12.260 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "45df5c6f-b124-46d1-9efb-62b78bc584ed", 2024-08-30T16:03:12.260 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:12.260 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:12.260 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:12.261 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:12.261 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:12.261 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:12.261 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:03:12.261 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 40, 2024-08-30T16:03:12.261 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:12.261 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:12.261 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:12.261 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.261 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.261 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6813", 2024-08-30T16:03:12.261 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:03:12.262 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.262 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.262 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:12.262 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.262 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.262 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6814", 2024-08-30T16:03:12.262 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:03:12.262 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.262 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.262 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6816", 2024-08-30T16:03:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:03:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6815", 2024-08-30T16:03:12.263 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:03:12.264 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.264 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.264 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6813/68721277", 2024-08-30T16:03:12.264 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6814/68721277", 2024-08-30T16:03:12.264 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6816/68721277", 2024-08-30T16:03:12.264 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6815/68721277", 2024-08-30T16:03:12.264 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:12.264 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:12.264 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:12.264 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:12.264 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.264 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:03:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_xinfo": [ 2024-08-30T16:03:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-08-30T16:03:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.911895+0000", 2024-08-30T16:03:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-08-30T16:03:12.265 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:04.786543+0000", 2024-08-30T16:03:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-08-30T16:03:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:12.266 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:03:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:04.197952+0000", 2024-08-30T16:03:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-08-30T16:03:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.680847+0000", 2024-08-30T16:03:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:12.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-08-30T16:03:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:03:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:01.937600+0000", 2024-08-30T16:03:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-08-30T16:03:12.268 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:02.082761+0000", 2024-08-30T16:03:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-08-30T16:03:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:03:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.401025+0000", 2024-08-30T16:03:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-08-30T16:03:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.129371+0000", 2024-08-30T16:03:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:12.270 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap": [], 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_items": [], 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_primaries": [], 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_temp": [], 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_temp": [], 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklist": {}, 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "range_blocklist": {}, 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profiles": { 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "default": { 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-08-30T16:03:12.271 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van" 2024-08-30T16:03:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps_queue": [], 2024-08-30T16:03:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "new_removed_snaps": [], 2024-08-30T16:03:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "new_purged_snaps": [], 2024-08-30T16:03:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_node_flags": {}, 2024-08-30T16:03:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "device_class_flags": {}, 2024-08-30T16:03:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": { 2024-08-30T16:03:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_enabled": false, 2024-08-30T16:03:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_bucket_count": 0, 2024-08-30T16:03:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "degraded_stretch_mode": 0, 2024-08-30T16:03:12.272 INFO:teuthology.orchestra.run.smithi050.stdout: "recovering_stretch_mode": 0, 2024-08-30T16:03:12.273 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_bucket": 0 2024-08-30T16:03:12.273 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:12.273 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:12.273 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:12.274 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:12.275 INFO:tasks.thrashosds.thrasher:j is {'epoch': 66, 'fsid': '9d72e2e2-0554-42af-8e62-9b1a3a804112', 'created': '2024-08-30T15:58:20.129410+0000', 'modified': '2024-08-30T16:03:09.195776+0000', 'last_up_change': '2024-08-30T15:59:19.280372+0000', 'last_in_change': '2024-08-30T16:03:05.847464+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T15:59:08.355540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-08-30T15:59:21.696796+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.96999979019165, 'score_stable': 4.96999979019165, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 3.130000114440918, 'raw_score_stable': 3.130000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 60, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6812', 'nonce': 1957399494}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6813', 'nonce': 1957399494}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 1957399494}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6814', 'nonce': 1957399494}]}, 'public_addr': '172.21.15.50:6812/1957399494', 'cluster_addr': '172.21.15.50:6813/1957399494', 'heartbeat_back_addr': '172.21.15.50:6815/1957399494', 'heartbeat_front_addr': '172.21.15.50:6814/1957399494', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0fe11a1c-4326-4bfd-a2a4-bff14023f301', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6808', 'nonce': 3208178756}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6809', 'nonce': 3208178756}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 3208178756}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6810', 'nonce': 3208178756}]}, 'public_addr': '172.21.15.50:6808/3208178756', 'cluster_addr': '172.21.15.50:6809/3208178756', 'heartbeat_back_addr': '172.21.15.50:6811/3208178756', 'heartbeat_front_addr': '172.21.15.50:6810/3208178756', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a9d5354b-3600-41ed-80c5-a3439da7aca6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6804', 'nonce': 287018800}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6805', 'nonce': 287018800}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 287018800}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6806', 'nonce': 287018800}]}, 'public_addr': '172.21.15.50:6804/287018800', 'cluster_addr': '172.21.15.50:6805/287018800', 'heartbeat_back_addr': '172.21.15.50:6807/287018800', 'heartbeat_front_addr': '172.21.15.50:6806/287018800', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '78bcc398-a7e4-4efc-9836-1dfb94407c9c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 64, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6800', 'nonce': 1140717714}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6801', 'nonce': 1140717714}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 1140717714}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6802', 'nonce': 1140717714}]}, 'public_addr': '172.21.15.50:6800/1140717714', 'cluster_addr': '172.21.15.50:6801/1140717714', 'heartbeat_back_addr': '172.21.15.50:6803/1140717714', 'heartbeat_front_addr': '172.21.15.50:6802/1140717714', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '53a1060b-7985-4c2c-820e-64b158056113', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 1468073345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6806', 'nonce': 1468073345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6808', 'nonce': 1468073345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 1468073345}]}, 'public_addr': '172.21.15.94:6805/1468073345', 'cluster_addr': '172.21.15.94:6806/1468073345', 'heartbeat_back_addr': '172.21.15.94:6808/1468073345', 'heartbeat_front_addr': '172.21.15.94:6807/1468073345', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '853a03f7-227d-4f98-9e5c-c29896820746', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 64, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 3876025616}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6810', 'nonce': 3876025616}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6812', 'nonce': 3876025616}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 3876025616}]}, 'public_addr': '172.21.15.94:6809/3876025616', 'cluster_addr': '172.21.15.94:6810/3876025616', 'heartbeat_back_addr': '172.21.15.94:6812/3876025616', 'heartbeat_front_addr': '172.21.15.94:6811/3876025616', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bc978060-bf28-4027-8bba-7d6d8659bfc5', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 128519122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6802', 'nonce': 128519122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6804', 'nonce': 128519122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 128519122}]}, 'public_addr': '172.21.15.94:6801/128519122', 'cluster_addr': '172.21.15.94:6802/128519122', 'heartbeat_back_addr': '172.21.15.94:6804/128519122', 'heartbeat_front_addr': '172.21.15.94:6803/128519122', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '45df5c6f-b124-46d1-9efb-62b78bc584ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 68721277}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6814', 'nonce': 68721277}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6816', 'nonce': 68721277}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 68721277}]}, 'public_addr': '172.21.15.94:6813/68721277', 'cluster_addr': '172.21.15.94:6814/68721277', 'heartbeat_back_addr': '172.21.15.94:6816/68721277', 'heartbeat_front_addr': '172.21.15.94:6815/68721277', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.911895+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:04.786543+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:04.197952+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.680847+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:01.937600+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:02.082761+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.401025+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.129371+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T16:03:12.275 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-08-30T16:03:12.275 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:12.273+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:12.308 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:03:12.308 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:03:12.365 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:12.365 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:12.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:12.375 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:12.373+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:12.412 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:12.412 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:12.476 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:12.475+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:12.545 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:12.545 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:12.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:12.577 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:12.575+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:12.678 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:12.677+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:12.683 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:12.683 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:12.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:12.778 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:12.777+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:12.861 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:12.861 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:12.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:12.914 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:12.913+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:12.981 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:12.979+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:13.001 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:13.001 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:13.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:13.081 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:13.080+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:13.179 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:13.179 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:13.182 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:13.180+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:13.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:13.283 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:13.282+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:13.318 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:13.318 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:13.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:13.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '7 of 30' 2024-08-30T16:03:13.418 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:13.415+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-08-30T16:03:13.485 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:13.483+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:13.522 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:13.522 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:13.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:13.586 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:13.584+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:13.695 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:13.694+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:13.707 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:13.707 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:13.716 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:13.787 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:13.786+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:13.852 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:13.852 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:13.862 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:13.906 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:13.904+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:13.986 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:13.986 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:13.989 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:13.988+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:13.995 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:14.091 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:14.089+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:14.158 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:14.158 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:14.168 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:14.192 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:14.190+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:14.291 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:14.292 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:14.292 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:14.290+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:14.300 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:14.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '8 of 30' 2024-08-30T16:03:14.393 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:14.391+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:14.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:03:14.464 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:14.465 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:14.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:14.515 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:14.513+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:14.595 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:14.593+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:14.639 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:14.639 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:14.649 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:03:14.696 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:14.694+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:14.709 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:14.710 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:14.777 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:14.777 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:14.786 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:03:14.809 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:14.809 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:14.830 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:14.828+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:14.909 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:14.909 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:14.913 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:14.913 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:14.913 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:14.912+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:14.917 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:03:15.002 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:15.000+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:15.022 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:15.022 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:15.082 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:15.082 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:15.091 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:15.124 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:15.124 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:15.150 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:15.148+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:15.201 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:15.199+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:15.209 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:15.209 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:15.268 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:15.269 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:15.281 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:15.305 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:03:15.306 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:03:15.322 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:15.320+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:15.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '9 of 30' 2024-08-30T16:03:15.389 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:15.390 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:15.402 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:15.400+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:15.445 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:15.445 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:15.445 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:15.445 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:15.445 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:15.454 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:15.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-08-30T16:03:15.527 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:15.526+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:15.604 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:15.602+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:15.627 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 227.247860135, 2024-08-30T16:03:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:03:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:15.628 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:03:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:03:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:03:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:03:15.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:03:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:03:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:03:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:03:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:03:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:15.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:03:15.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:03:15.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:15.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:03:15.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:03:15.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:15.632 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:15.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:15.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.633 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:03:15.633 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:15.633 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 107.13486401900001, 2024-08-30T16:03:15.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:03:15.633 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:15.633 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:15.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:15.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:15.633 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:15.633 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:03:15.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.633 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:15.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:15.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:15.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:15.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:15.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:15.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:15.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:15.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:03:15.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:15.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:15.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:03:15.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:03:15.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:15.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:03:15.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:03:15.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:15.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:03:15.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:03:15.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:15.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:03:15.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:03:15.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:15.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:15.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:15.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:03:15.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:03:15.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:15.636 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:15.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:15.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:15.637 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:15.637 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:15.638 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:15.705 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:15.704+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:15.765 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:15.765 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:15.765 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:15.765 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:15.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:15.815 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:15.814+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:15.908 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:15.906+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:15.977 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:15.977 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:15.978 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:15.978 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:15.978 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:15.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:16.009 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:16.007+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:16.110 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:16.109+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:16.122 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:16.122 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:16.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:16.122 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:03:16.122 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:16.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:16.216 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:16.214+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:16.299 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:16.299 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:16.299 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:16.300 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:16.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:16.312 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:16.310+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:16.396 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-08-30T16:03:16.413 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:16.411+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:16.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '10 of 30' 2024-08-30T16:03:16.483 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:16.483 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:16.483 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:16.483 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:16.483 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:16.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:16.514 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:16.512+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:16.615 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:16.614+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:16.700 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:16.700 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:16.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:16.700 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:16.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.700 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:16.700 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:16.701 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 235.42544168699999, 2024-08-30T16:03:16.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:03:16.701 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:16.701 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:16.701 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:16.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:16.701 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:16.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:16.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:16.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:16.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:16.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:16.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:03:16.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:03:16.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:16.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:03:16.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:03:16.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:16.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:03:16.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:03:16.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:16.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:03:16.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:03:16.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:16.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:03:16.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:03:16.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:16.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:03:16.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:03:16.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:16.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:03:16.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:03:16.704 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:16.704 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:16.704 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:16.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.705 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:16.705 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:16.705 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 235.15251432599999, 2024-08-30T16:03:16.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:03:16.705 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:16.705 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:16.705 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:16.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:16.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:16.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:16.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:16.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:16.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:16.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:16.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:03:16.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:03:16.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:16.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:03:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:03:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:03:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:03:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:03:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:03:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.708 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:16.708 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:03:16.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:03:16.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.708 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:16.708 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:03:16.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:03:16.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:16.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:16.708 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:16.708 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:03:16.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:03:16.709 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:16.709 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:16.709 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:16.709 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:16.709 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:16.709 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:16.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:16.754 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:16.753+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:16.817 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:16.816+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:16.898 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:16.898 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:16.898 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:16.898 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:16.908 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:16.950 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:16.948+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:17.019 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:17.017+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:17.089 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:17.089 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:17.089 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:17.090 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:17.090 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:17.098 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:17.140 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:17.139+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:17.222 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:17.220+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:17.268 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:17.269 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:17.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:17.269 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:17.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.269 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:03:17.269 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:17.269 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 231.891296474, 2024-08-30T16:03:17.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:03:17.269 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:17.269 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:17.269 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:17.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:17.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:17.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:17.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:17.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:17.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:03:17.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:03:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:03:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:03:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:03:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:03:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:17.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:03:17.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:03:17.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.272 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:17.272 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:03:17.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:03:17.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.272 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:17.272 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:03:17.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:03:17.273 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.273 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:17.273 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:03:17.273 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:03:17.273 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.273 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:17.273 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:03:17.273 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:03:17.273 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.273 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:17.273 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.274 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.274 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:17.274 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:17.276 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7] out_osds: [6, 4, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:03:17.276 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:03:17.276 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-08-30T16:03:17.276 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T16:03:17.277 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T16:03:17.277 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-08-30T16:03:17.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:17.326 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:17.323+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:17.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:03:17.406 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:17.407 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:17.407 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:17.407 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:17.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:17.418 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:17.418 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-30T16:03:17.418 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:17.426 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:17.424+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:17.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '11 of 30' 2024-08-30T16:03:17.527 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:17.525+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:17.570 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:40.175 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:17.579 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:40.185 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:17.613 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:40.218 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:17.615 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:17.615 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:17.615 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:17.615 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:17.615 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:17.628 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:17.626+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:17.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:17.639 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:59.029 42904== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:17.659 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:59.049 42904== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:17.697 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:59.087 42904== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:17.729 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:17.727+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:17.782 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:17.783 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:17.853 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:17.853 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:17.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:17.853 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:17.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.854 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:17.854 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:17.854 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 233.389910617, 2024-08-30T16:03:17.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:03:17.854 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:17.854 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:17.854 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:17.854 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:17.854 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:17.854 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:17.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:17.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:03:17.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:03:17.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:17.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:03:17.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:03:17.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:17.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:03:17.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:03:17.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:17.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:03:17.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:03:17.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:17.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:03:17.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:03:17.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:03:17.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:03:17.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:03:17.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:17.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:03:17.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:03:17.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:17.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:03:17.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:03:17.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:03:17.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:03:17.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:03:17.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:17.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:03:17.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:03:17.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:17.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:03:17.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:03:17.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:17.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:03:17.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:03:17.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:17.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:03:17.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:03:17.861 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.861 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:17.861 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.861 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:03:17.861 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:17.861 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 232.27404428200001, 2024-08-30T16:03:17.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:03:17.862 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:17.862 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:17.862 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:17.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:17.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:17.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:17.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:17.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:17.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:03:17.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:17.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:17.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:03:17.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:03:17.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:03:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:03:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:03:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:03:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:03:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:03:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:17.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:03:17.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:03:17.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:17.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:03:17.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:03:17.865 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.865 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:17.865 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.866 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:17.866 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:17.866 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 232.214452784, 2024-08-30T16:03:17.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:03:17.866 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:17.866 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:17.866 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:17.866 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:17.866 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:17.866 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:03:17.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.866 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:17.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:17.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:17.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:17.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:17.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:17.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:03:17.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:03:17.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:17.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:03:17.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:03:17.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:17.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:03:17.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:03:17.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:17.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:03:17.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:03:17.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:17.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:03:17.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:03:17.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:17.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:03:17.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:03:17.869 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.869 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:03:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 232.18775409700001, 2024-08-30T16:03:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:03:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:17.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.871 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:17.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:17.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:17.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:17.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:17.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:03:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:03:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:03:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:03:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:03:17.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:03:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:03:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:03:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:03:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:03:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:03:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 231.320137569, 2024-08-30T16:03:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:03:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:17.875 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:17.875 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:17.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.875 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:17.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:17.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:17.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:17.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:17.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:17.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:03:17.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:17.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:17.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:03:17.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:03:17.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:17.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:03:17.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:03:17.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:17.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:03:17.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:03:17.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:17.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:03:17.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:03:17.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:17.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:03:17.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:03:17.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:17.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:03:17.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:03:17.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:17.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:03:17.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:03:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:03:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:03:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:03:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:03:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:17.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:03:17.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:03:17.880 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.880 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:17.880 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.880 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:03:17.880 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:17.880 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 231.16089049000001, 2024-08-30T16:03:17.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:03:17.880 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:17.880 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:17.880 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:17.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:17.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:17.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:17.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:17.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:17.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:03:17.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:03:17.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:17.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:03:17.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:03:17.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:17.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:03:17.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:03:17.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:17.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:03:17.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:03:17.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:17.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:03:17.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:03:17.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:17.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:03:17.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:03:17.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:17.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:03:17.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:03:17.884 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.884 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:17.884 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.884 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:17.884 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:17.884 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 230.884716424, 2024-08-30T16:03:17.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:03:17.885 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:17.885 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:17.885 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:17.885 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:17.885 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:17.885 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:17.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.885 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:17.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:17.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:17.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:03:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:03:17.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:03:17.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:17.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:03:17.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:03:17.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:17.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:03:17.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:03:17.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:17.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:03:17.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:03:17.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:17.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:03:17.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:03:17.888 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.888 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:17.888 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.888 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:17.889 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:17.889 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 230.226424845, 2024-08-30T16:03:17.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:03:17.889 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:17.889 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:17.889 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:17.889 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:17.889 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:17.889 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:03:17.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.889 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:17.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:17.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:17.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:17.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:17.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:17.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:03:17.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:17.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:17.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:03:17.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:03:17.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:17.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:03:17.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:03:17.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:17.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:03:17.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:03:17.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:17.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:03:17.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:03:17.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:17.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:03:17.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:03:17.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:17.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:03:17.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:03:17.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:17.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:03:17.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:03:17.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:17.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:03:17.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:03:17.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:17.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:03:17.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:03:17.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:17.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:03:17.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:03:17.894 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.894 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:17.894 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.895 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:03:17.895 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:17.895 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 230.13051156099999, 2024-08-30T16:03:17.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:03:17.895 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:17.895 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:17.895 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:17.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:17.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:17.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:17.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:17.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:17.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:03:17.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:17.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:17.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:03:17.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:03:17.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:17.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:03:17.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:03:17.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:17.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:03:17.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:03:17.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:17.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:03:17.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:03:17.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:17.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:03:17.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:03:17.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:17.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:03:17.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:03:17.898 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.898 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:17.899 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.899 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:17.899 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:17.899 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 170.87915104999999, 2024-08-30T16:03:17.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:03:17.899 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:17.899 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:17.899 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:17.900 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:17.900 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:17.900 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:17.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.900 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:17.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:17.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:17.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:17.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:17.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:17.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:03:17.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:03:17.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:17.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:03:17.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:03:17.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:17.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:03:17.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:03:17.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:17.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:03:17.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:03:17.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:17.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:03:17.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:03:17.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:17.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:03:17.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:03:17.903 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.904 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:17.904 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.904 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:03:17.904 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:17.904 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 110.870046363, 2024-08-30T16:03:17.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:03:17.904 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:17.904 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:17.904 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:17.904 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:17.904 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:17.905 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:17.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.905 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:17.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:17.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:17.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:17.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:17.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:17.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:17.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:03:17.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:03:17.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:17.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:03:17.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:03:17.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:17.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:03:17.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:03:17.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:17.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:03:17.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:03:17.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:17.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:03:17.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:03:17.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:17.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:17.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:17.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:03:17.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:03:17.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.908 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:17.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:17.908 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:17.908 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:17.909 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:17.909 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:17.910 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:17.910 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:17.889+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:17.954 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:17.952+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:17.982 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:17.982 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:18.032 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:18.030+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:18.045 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:18.045 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:18.045 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:18.045 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:18.055 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:18.073 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:18.073 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:18.140 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:18.138+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:18.158 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:18.158 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:18.223 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:18.223 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:18.233 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:18.236 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:18.234+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:18.263 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:18.264 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:18.333 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:18.332+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:18.371 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:03:18.371 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:03:18.401 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:18.402 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:18.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-08-30T16:03:18.411 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:18.455 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:18.456 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:18.456 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:18.454+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:18.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '12 of 30' 2024-08-30T16:03:18.535 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:18.534+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:18.539 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:18.539 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:18.548 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:18.636 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:18.635+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:18.712 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:18.713 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:18.722 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:18.737 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:18.735+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:18.838 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:18.836+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:18.849 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:18.849 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:18.858 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:03:18.939 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:18.938+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:19.022 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:19.022 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:19.032 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:03:19.077 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:19.075+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:19.141 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:19.139+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:19.155 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:19.155 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:19.165 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:03:19.241 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:19.240+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:19.331 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:19.331 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:19.342 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:19.342 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:19.341+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:19.443 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:19.441+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:19.466 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:19.466 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:19.476 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:19.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '13 of 30' 2024-08-30T16:03:19.560 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:19.558+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:19.617 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-08-30T16:03:19.640 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:19.640 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:19.644 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:19.643+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:19.649 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:19.746 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:19.744+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:19.813 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:19.813 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:19.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:19.846 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:19.845+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:19.948 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:19.946+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:19.951 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:19.951 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:19.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:20.048 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:20.047+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:20.131 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:20.132 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:20.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:20.149 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:20.148+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:20.250 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:20.248+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:20.311 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:20.312 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:20.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:20.364 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:20.363+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:20.452 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:20.451+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:20.456 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:20.456 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:20.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:20.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:03:20.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '14 of 30' 2024-08-30T16:03:20.573 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:20.570+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:20.654 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:20.652+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:20.658 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:20.658 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:20.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:20.755 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:20.753+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:20.779 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:20.779 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:20.844 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:20.844 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:20.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:20.856 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:20.854+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:20.909 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:20.909 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:20.958 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:20.957+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:21.043 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:21.043 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:21.047 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:21.047 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:21.047 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:21.047 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:21.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:21.058 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:21.057+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:21.146 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:21.147 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:21.159 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:21.157+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:21.186 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:21.187 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:21.187 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:21.187 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:21.187 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:21.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:21.233 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:21.233 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:21.260 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:21.258+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:21.330 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:21.330 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:21.334 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:21.334 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:21.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:21.335 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:21.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.335 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:03:21.335 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:21.335 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 235.95751668599999, 2024-08-30T16:03:21.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:03:21.335 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:21.335 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:21.335 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:21.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:21.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:21.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:21.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:21.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:21.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:03:21.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:03:21.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:21.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:03:21.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:03:21.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:21.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:03:21.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:03:21.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:21.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:03:21.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:03:21.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:21.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:03:21.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:03:21.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:21.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:03:21.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:03:21.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:21.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:03:21.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:03:21.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:21.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:03:21.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:03:21.339 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.339 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:21.339 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.339 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.339 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:21.339 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:21.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:21.361 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:21.359+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:21.411 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:03:21.411 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:03:21.494 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:21.494 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:21.495 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:21.492+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:21.515 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:21.516 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:21.516 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:21.516 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:21.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:21.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '15 of 30' 2024-08-30T16:03:21.587 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:21.585+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:21.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-08-30T16:03:21.663 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:21.662+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:21.728 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:21.728 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:21.728 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:21.728 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:21.728 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:21.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:21.764 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:21.763+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:21.865 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:21.863+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:21.898 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:21.898 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:21.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:21.898 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:21.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.899 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:21.899 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:21.899 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 237.44010648, 2024-08-30T16:03:21.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:03:21.899 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:21.900 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:21.900 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:21.900 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:21.900 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:21.900 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:21.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.901 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:21.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:21.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:21.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:21.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:21.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:21.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:03:21.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:03:21.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:21.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:03:21.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:03:21.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:21.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:03:21.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:03:21.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:21.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:03:21.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:03:21.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:21.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:03:21.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:03:21.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:03:21.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:03:21.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:03:21.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:21.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:03:21.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:03:21.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:21.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:03:21.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:03:21.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:03:21.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:03:21.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:03:21.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:21.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:03:21.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:03:21.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:21.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:03:21.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:03:21.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:21.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:03:21.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:03:21.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:21.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:03:21.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:03:21.907 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.907 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:21.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.908 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:03:21.908 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:21.908 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 236.324240145, 2024-08-30T16:03:21.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:03:21.908 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:21.909 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:21.909 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:21.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:21.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:21.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:21.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:21.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:21.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:03:21.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:21.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:21.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:03:21.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:03:21.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:21.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:03:21.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:03:21.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:21.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:03:21.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:03:21.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:21.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:03:21.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:03:21.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:21.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:03:21.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:03:21.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:21.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:03:21.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:03:21.912 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.912 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:21.913 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.913 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:21.913 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:21.913 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 236.264648647, 2024-08-30T16:03:21.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:03:21.913 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:21.913 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:21.913 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:21.913 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:21.913 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:21.914 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:03:21.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.914 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:21.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:21.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:21.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:21.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:21.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:21.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:03:21.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:03:21.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:21.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:03:21.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:03:21.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:21.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:03:21.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:03:21.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:21.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:03:21.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:03:21.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:21.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:03:21.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:03:21.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:21.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:03:21.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:03:21.917 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.917 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:21.918 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.918 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:03:21.918 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:21.918 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 236.23794996000001, 2024-08-30T16:03:21.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:03:21.918 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:21.919 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:21.919 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:21.919 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:21.919 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:21.919 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:21.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.919 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:21.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:21.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:21.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:21.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:21.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:21.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:03:21.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:21.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:21.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:03:21.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:03:21.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:21.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:03:21.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:03:21.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:21.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:03:21.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:03:21.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:21.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:03:21.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:03:21.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:21.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:03:21.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:03:21.923 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.923 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:21.923 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.923 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:21.923 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:21.923 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 235.370333432, 2024-08-30T16:03:21.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:03:21.924 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:21.924 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:21.924 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:21.924 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:21.924 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:21.924 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:21.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.924 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:21.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:21.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:21.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:21.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:21.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:21.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:03:21.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:21.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:21.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:03:21.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:03:21.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:21.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:03:21.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:03:21.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:21.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:03:21.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:03:21.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:21.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:03:21.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:03:21.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:21.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:03:21.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:03:21.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:21.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:03:21.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:03:21.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:21.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:03:21.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:03:21.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:21.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:03:21.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:03:21.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:21.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:03:21.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:03:21.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:21.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:03:21.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:03:21.929 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.930 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:21.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.930 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:03:21.930 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:21.930 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 235.21108635300001, 2024-08-30T16:03:21.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:03:21.930 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:21.930 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:21.930 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:21.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:21.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:21.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:21.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:21.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:21.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:03:21.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:03:21.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:21.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:03:21.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:03:21.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:21.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:03:21.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:03:21.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:21.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:03:21.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:03:21.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:21.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:03:21.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:03:21.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:21.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:03:21.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:03:21.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:21.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:03:21.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:03:21.934 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.934 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:21.934 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.935 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:21.935 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:21.935 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 234.934912287, 2024-08-30T16:03:21.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:03:21.935 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:21.935 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:21.935 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:21.935 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:21.935 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:21.935 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:21.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.935 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:21.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:21.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:21.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:21.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:21.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:21.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:03:21.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:21.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:21.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:03:21.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:03:21.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:21.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:03:21.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:03:21.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:21.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:03:21.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:03:21.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:21.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:03:21.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:03:21.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:21.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:03:21.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:03:21.939 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.939 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:21.939 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.939 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:21.939 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:21.939 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 234.276620708, 2024-08-30T16:03:21.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:03:21.939 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:21.939 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:21.939 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:21.940 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:21.940 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:21.940 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:03:21.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.940 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:21.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:21.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:21.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:21.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:21.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:21.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:03:21.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:21.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:21.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:03:21.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:03:21.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:21.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:03:21.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:03:21.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:21.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:03:21.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:03:21.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:21.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:03:21.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:03:21.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:21.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:03:21.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:03:21.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:21.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:03:21.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:03:21.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:21.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:03:21.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:03:21.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:21.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:03:21.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:03:21.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:21.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:03:21.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:03:21.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:21.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:03:21.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:03:21.945 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.945 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:21.945 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.946 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:03:21.946 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:21.946 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 234.18070742399999, 2024-08-30T16:03:21.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:03:21.946 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:21.946 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:21.946 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:21.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:21.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:21.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:03:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:03:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:03:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:03:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:03:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:03:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:03:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:03:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:03:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:03:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:03:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:21.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:03:21.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:03:21.950 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.950 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:21.950 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.950 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:21.951 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:21.951 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 174.92934691299999, 2024-08-30T16:03:21.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:03:21.951 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:21.951 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:21.951 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:21.951 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:21.951 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:21.951 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:21.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.951 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:21.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:21.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:21.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:21.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:21.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:21.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:03:21.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:03:21.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:21.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:03:21.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:03:21.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:21.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:03:21.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:03:21.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:21.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:03:21.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:03:21.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:21.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:03:21.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:03:21.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:21.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:03:21.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:03:21.954 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.955 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:21.955 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.955 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:03:21.955 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:21.955 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 114.920242226, 2024-08-30T16:03:21.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:03:21.955 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:21.955 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:21.955 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:21.955 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:21.955 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:21.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:03:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:03:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:03:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:03:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:21.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:03:21.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:03:21.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:21.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:03:21.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:03:21.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:21.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:03:21.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:03:21.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:21.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:21.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:21.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:03:21.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:03:21.959 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.959 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:21.959 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.959 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:21.959 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:21.959 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:21.961 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:22.003 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:22.001+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:22.078 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:22.077+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:22.092 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:22.093 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:22.093 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:22.093 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:22.101 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:22.168 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:22.166+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:22.265 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:22.265 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:22.266 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:22.269 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:22.267+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:22.274 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:22.370 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:22.369+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:22.428 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7] out_osds: [6, 4, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:03:22.428 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:03:22.440 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:22.440 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:22.440 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:22.440 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:03:22.440 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:22.449 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:22.493 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:22.491+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:22.573 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:22.571+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:22.573 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:22.573 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:22.573 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:22.573 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:22.583 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:22.652 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-08-30T16:03:22.673 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:22.671+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:22.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '16 of 30' 2024-08-30T16:03:22.746 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:22.747 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:22.747 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:22.747 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:22.747 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:22.756 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:22.798 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:22.796+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:22.874 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:22.873+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:22.967 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:22.967 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 252.106625729, 2024-08-30T16:03:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:03:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:03:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:22.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:03:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:03:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:03:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:03:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:03:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:03:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:03:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:03:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:03:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:03:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.972 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:03:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:03:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:03:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:03:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:03:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:03:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:03:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:03:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:03:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:22.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:03:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:03:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:03:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:03:22.975 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 251.19020970899999, 2024-08-30T16:03:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:03:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:22.976 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:03:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:03:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:22.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:03:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:03:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:03:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:03:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:03:22.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:03:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:03:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:03:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:03:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:03:22.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:03:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:03:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:03:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:03:22.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:03:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:03:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.982 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 251.18850515599999, 2024-08-30T16:03:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:03:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:03:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:22.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:03:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:03:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:03:22.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:03:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:03:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:03:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:03:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:03:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:03:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:03:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:03:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:03:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:03:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:03:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:03:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:03:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:22.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:03:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:03:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 250.76905134899999, 2024-08-30T16:03:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:03:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:22.989 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:03:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:03:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:03:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:03:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:03:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:03:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:03:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:22.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:03:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:03:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:03:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:03:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:22.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:03:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:03:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:03:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:03:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:03:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:03:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:03:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:03:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 250.75216317499999, 2024-08-30T16:03:22.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:03:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:03:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:22.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:03:22.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:03:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:03:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:03:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:03:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:03:22.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:03:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:03:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:03:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:03:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:22.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:03:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:03:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:03:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:03:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:03:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:03:23.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:03:23.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:03:23.001 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.001 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.001 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.001 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:23.001 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:23.001 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 250.73916310600001, 2024-08-30T16:03:23.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:03:23.002 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.002 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.002 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.002 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.002 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.002 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:03:23.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.002 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:23.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:23.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:03:23.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:03:23.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:03:23.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:03:23.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:03:23.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:03:23.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:03:23.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:03:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:03:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:03:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:03:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:03:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:03:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:03:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:03:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:03:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 250.73667081400001, 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.007 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:03:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:23.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:03:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:03:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:03:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:03:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:03:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:03:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:03:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:03:23.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:03:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:03:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:03:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:03:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:03:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:03:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:03:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:03:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:03:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.012 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 250.73591336600001, 2024-08-30T16:03:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:03:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:03:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:03:23.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:03:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:03:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:03:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:03:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:03:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:03:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:03:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:03:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:03:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:03:23.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:03:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:03:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:03:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:03:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:03:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:03:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:03:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 250.73518091599999, 2024-08-30T16:03:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:03:23.018 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:03:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:03:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:03:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:03:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:03:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:03:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:03:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:03:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:03:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:03:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:03:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:03:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:03:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:03:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:03:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:03:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:03:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:03:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.023 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 250.73264211700001, 2024-08-30T16:03:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:03:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:03:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:03:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.025 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:03:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:03:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:03:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:03:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:03:23.026 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:03:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:03:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:03:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:03:23.027 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:03:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:03:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:03:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:03:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:03:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.028 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:03:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:03:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 250.73189411600001, 2024-08-30T16:03:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:03:23.029 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:03:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.030 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:03:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:03:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:03:23.031 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:03:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:03:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:03:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:03:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:03:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.032 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:03:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:03:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:03:23.033 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:03:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:03:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:03:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:03:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:03:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.034 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:03:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:03:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 249.95977764400001, 2024-08-30T16:03:23.035 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:03:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:03:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.036 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:03:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:03:23.037 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:03:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:03:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:03:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:03:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.038 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:03:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:03:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:03:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:03:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:03:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:03:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.039 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:03:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:03:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:03:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:03:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:03:23.040 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:03:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 249.95772049799999, 2024-08-30T16:03:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:03:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.041 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:03:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.042 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:03:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:03:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:03:23.043 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:03:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:03:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:03:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:03:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:03:23.044 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:03:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:03:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:03:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:03:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:03:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.045 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:03:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:03:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:03:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:03:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:23.046 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 249.69477599800001, 2024-08-30T16:03:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:03:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:03:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:23.047 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:03:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:03:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.048 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:03:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:03:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:03:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:03:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:03:23.049 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:03:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:03:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:03:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:03:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:03:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.050 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:03:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:03:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:03:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:03:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:03:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:03:23.051 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 249.69368449500001, 2024-08-30T16:03:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:03:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.052 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:03:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:03:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:03:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:03:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:03:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:03:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:03:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:03:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:03:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:03:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:03:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:03:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:03:23.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:03:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:03:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:03:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:03:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:03:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.057 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 249.69293490999999, 2024-08-30T16:03:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:03:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:03:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.058 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:03:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:03:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:03:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:03:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:03:23.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:03:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:03:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:03:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:03:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:03:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:03:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:03:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:03:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:03:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:03:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:03:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:03:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.063 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 249.659203378, 2024-08-30T16:03:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:03:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:03:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:03:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:03:23.065 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:03:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:03:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:03:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:03:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:03:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:03:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:03:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:03:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:03:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:03:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:03:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:03:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:03:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:03:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:03:23.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:03:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 249.65867736999999, 2024-08-30T16:03:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:03:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.069 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:03:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:23.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:03:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:03:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:03:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:03:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:03:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:03:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:03:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:03:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:03:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:03:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:03:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:03:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:03:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:03:23.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:03:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:03:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:03:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 249.65822180699999, 2024-08-30T16:03:23.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:03:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:03:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:03:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:03:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:03:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:03:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:03:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:03:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:03:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:03:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:03:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:03:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:03:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:03:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:03:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:03:23.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:03:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:03:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:03:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:03:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:03:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 244.98770649100001, 2024-08-30T16:03:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:03:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:03:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:03:23.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:03:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:03:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:03:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:03:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:03:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:03:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:03:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:03:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:03:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:03:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:03:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:03:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:03:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:03:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:03:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:03:23.084 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.085 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.085 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:23.086 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:03:23.087 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:23.085+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:23.188 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:23.186+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:23.252 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:23.252 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:23.252 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:23.252 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:23.261 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:03:23.303 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:23.301+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:23.389 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:23.387+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:23.424 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:23.424 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:23.425 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:23.425 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:23.433 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:03:23.489 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:23.488+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:23.508 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:03:23.590 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:23.589+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:23.601 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:23.601 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:23.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:23.601 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:23.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.601 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:03:23.601 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:03:23.601 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 237.07112460799999, 2024-08-30T16:03:23.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:03:23.602 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.602 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.602 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:03:23.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:03:23.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:03:23.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:03:23.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:03:23.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:03:23.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:03:23.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:03:23.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:03:23.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:03:23.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:03:23.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:03:23.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:03:23.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:03:23.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.605 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:03:23.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:03:23.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:03:23.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:03:23.606 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.606 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.606 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.606 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:03:23.606 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:03:23.606 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 235.952676462, 2024-08-30T16:03:23.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:03:23.606 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:23.607 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:23.607 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:23.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:23.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:03:23.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:23.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:03:23.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:23.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:23.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:03:23.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:03:23.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:23.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:03:23.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:03:23.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:23.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:03:23.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:03:23.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:23.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:03:23.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:03:23.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:23.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:03:23.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:03:23.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:23.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:03:23.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:03:23.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:23.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:03:23.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:03:23.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:23.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:23.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:23.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:03:23.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:03:23.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.611 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.611 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:23.611 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:23.611 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:23.612 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:23.691 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:23.690+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:23.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '17 of 30' 2024-08-30T16:03:23.737 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:23.738 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:23.738 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:23.738 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:23.747 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:23.824 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:23.824 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:23.831 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:23.829+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:23.894 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:23.892+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:23.910 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:23.910 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:23.910 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:23.910 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:23.910 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:23.919 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:23.921 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:23.921 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:23.994 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:23.993+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:24.010 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:24.010 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:24.088 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:24.088 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:24.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:24.088 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:24.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.088 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:24.088 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:24.088 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 235.70822071500001, 2024-08-30T16:03:24.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:03:24.089 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:24.089 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:24.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:24.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:24.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:24.089 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:03:24.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.089 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:24.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:24.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:24.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:24.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:24.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:24.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:24.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:24.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:03:24.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:03:24.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:24.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:03:24.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:03:24.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:24.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:03:24.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:03:24.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:24.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:03:24.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:03:24.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:24.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:03:24.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:03:24.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:24.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:03:24.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:03:24.093 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:24.093 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:24.093 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:24.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.097 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:03:24.097 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:24.097 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 115.59522459900001, 2024-08-30T16:03:24.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:03:24.097 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:24.097 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:24.097 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:24.097 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:24.097 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:24.097 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:03:24.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.098 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:24.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:24.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:24.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:24.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:24.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:24.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:24.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:24.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:03:24.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:24.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:24.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:03:24.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:03:24.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:24.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:03:24.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:03:24.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:24.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:03:24.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:03:24.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:24.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:03:24.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:03:24.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:24.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:24.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:24.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:03:24.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:03:24.101 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:24.101 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:24.101 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:24.101 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:24.101 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:24.101 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:24.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:24.103 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:24.101+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:24.106 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:24.107 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:24.198 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:24.199 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:24.203 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:24.202+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:24.244 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:24.244 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:24.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:24.294 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:24.294 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:24.337 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:24.335+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:03:24.375 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:03:24.405 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:24.403+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:24.420 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:24.420 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:24.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:24.455 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:24.455 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:24.508 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:24.504+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:24.582 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:24.582 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:24.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:24.606 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:24.605+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:24.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-08-30T16:03:24.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '18 of 30' 2024-08-30T16:03:24.707 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:24.705+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:24.758 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:24.758 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:24.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:24.814 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:24.811+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:24.908 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:24.907+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:24.960 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:24.960 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:24.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:25.009 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:25.008+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:25.098 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:25.098 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:25.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:25.110 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:25.109+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:25.211 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:25.210+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:25.239 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:25.239 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:25.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:25.312 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:25.310+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:25.376 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:25.376 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:25.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:25.413 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:25.411+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:25.514 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:25.512+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:25.554 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:25.554 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:25.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:25.647 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:25.646+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:25.716 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:25.714+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:25.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '19 of 30' 2024-08-30T16:03:25.743 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:25.743 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:25.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:25.786 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-08-30T16:03:25.816 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:25.814+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:25.917 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:25.916+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:25.922 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:25.922 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:25.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:26.018 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:26.016+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:26.100 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:26.100 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:26.110 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:26.155 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:26.153+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:26.220 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:26.219+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:26.247 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:26.247 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:26.257 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:26.342 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:26.340+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:26.422 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:26.420+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:26.422 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:26.422 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:26.431 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:26.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:03:26.522 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:26.521+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:26.595 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:26.595 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:26.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:26.623 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:26.622+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:26.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '20 of 30' 2024-08-30T16:03:26.723 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:26.724 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:26.724 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:26.722+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:26.727 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:26.727 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:26.736 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:26.820 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:26.824 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:26.823+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:26.872 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:26.873 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:26.882 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:26.926 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:26.926 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:26.924+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:26.926 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:27.017 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:27.017 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:27.049 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:27.047+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:27.056 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:27.056 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:27.065 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:03:27.108 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:27.108 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:27.150 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:27.148+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:27.192 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:27.192 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:27.228 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:27.227+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:27.280 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:03:27.281 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:03:27.281 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:27.281 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:27.282 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:27.282 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:27.292 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:03:27.365 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:27.365 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:27.366 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:27.363+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:27.429 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7] out_osds: [6, 4, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:03:27.429 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:03:27.429 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-30T16:03:27.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-08-30T16:03:27.430 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:27.427+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:27.456 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:27.456 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:27.457 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:27.465 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:03:27.534 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:27.533+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:27.631 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:27.629+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:27.636 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:27.637 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:27.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:27.637 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:27.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.637 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:5 2.2 e27/24 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-08-30T16:03:27.637 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:03:27.637 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 241.10708684599999, 2024-08-30T16:03:27.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12670508999999999, 2024-08-30T16:03:27.637 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:27.637 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:27.637 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:27.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:27.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:03:27.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:27.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:27.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524785+0000", 2024-08-30T16:03:27.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:27.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:27.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.524774+0000", 2024-08-30T16:03:27.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99999 2024-08-30T16:03:27.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:27.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.531547+0000", 2024-08-30T16:03:27.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0067730580000000002 2024-08-30T16:03:27.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:27.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.532457+0000", 2024-08-30T16:03:27.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00090984799999999999 2024-08-30T16:03:27.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:27.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.540533+0000", 2024-08-30T16:03:27.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0080753460000000006 2024-08-30T16:03:27.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:27.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.547072+0000", 2024-08-30T16:03:27.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0065397550000000004 2024-08-30T16:03:27.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:27.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.555968+0000", 2024-08-30T16:03:27.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088959339999999994 2024-08-30T16:03:27.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:27.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.639793+0000", 2024-08-30T16:03:27.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083824299000000005 2024-08-30T16:03:27.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:27.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:26.651490+0000", 2024-08-30T16:03:27.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.011697093 2024-08-30T16:03:27.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:27.642 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:27.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:27.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.642 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_repop(client.4288.0:7 2.2 e28/24 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-08-30T16:03:27.642 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:03:27.642 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 239.9886387, 2024-08-30T16:03:27.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.037715775999999999, 2024-08-30T16:03:27.642 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:27.642 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:27.642 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:27.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:27.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:03:27.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:27.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:27.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643233+0000", 2024-08-30T16:03:27.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:27.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:27.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643222+0000", 2024-08-30T16:03:27.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:03:27.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:27.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643351+0000", 2024-08-30T16:03:27.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012892700000000001 2024-08-30T16:03:27.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:27.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643465+0000", 2024-08-30T16:03:27.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000114336 2024-08-30T16:03:27.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:27.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.643589+0000", 2024-08-30T16:03:27.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012374400000000001 2024-08-30T16:03:27.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:27.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644581+0000", 2024-08-30T16:03:27.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00099196500000000003 2024-08-30T16:03:27.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:27.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.644960+0000", 2024-08-30T16:03:27.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000379317 2024-08-30T16:03:27.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:27.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680483+0000", 2024-08-30T16:03:27.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035522140000000001 2024-08-30T16:03:27.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:27.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:27.646 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:27.646 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:27.680949+0000", 2024-08-30T16:03:27.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00046589199999999998 2024-08-30T16:03:27.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:27.646 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:27.647 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:27.647 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:27.647 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:27.647 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:27.648 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:27.733 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:27.731+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:27.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-08-30T16:03:27.821 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:27.821 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:27.821 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:27.821 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:27.830 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:27.846 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:27.845+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:27.934 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:27.933+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:27.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '21 of 30' 2024-08-30T16:03:27.954 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:27.955 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:27.955 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:27.955 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:27.955 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:27.964 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:28.035 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:28.034+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:28.091 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:28.092 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:28.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:28.092 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:28.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.092 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:28.092 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:28.093 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 239.71195230500001, 2024-08-30T16:03:28.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:03:28.093 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:28.093 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:28.093 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:28.093 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:28.093 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:28.093 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:03:28.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.093 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:28.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:28.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:28.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:28.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:28.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:28.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:28.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:28.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:03:28.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:03:28.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:28.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:03:28.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:03:28.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:28.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:03:28.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:03:28.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:28.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:03:28.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:03:28.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:28.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:03:28.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:03:28.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:28.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:03:28.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:03:28.097 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:28.097 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:28.097 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:28.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.097 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:03:28.097 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:28.097 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 119.59895618900001, 2024-08-30T16:03:28.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:03:28.097 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:28.097 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:28.098 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:28.098 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:28.098 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:28.098 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:03:28.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.098 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:28.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:28.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:28.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:28.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:28.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:28.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:28.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:28.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:03:28.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:28.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:28.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:03:28.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:03:28.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:28.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:03:28.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:03:28.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:28.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:03:28.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:03:28.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:28.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:03:28.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:03:28.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:28.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:28.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:28.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:03:28.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:03:28.101 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:28.101 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:28.101 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:28.101 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:28.101 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:28.102 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:28.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:28.136 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:28.134+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:28.237 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:28.236+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:28.253 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:28.253 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:28.253 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:28.253 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:28.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:28.338 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:28.337+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:28.407 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:03:28.407 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:28.407 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 71, 2024-08-30T16:03:28.407 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "9d72e2e2-0554-42af-8e62-9b1a3a804112", 2024-08-30T16:03:28.407 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-08-30T15:58:20.129410+0000", 2024-08-30T16:03:28.407 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-08-30T16:03:25.685938+0000", 2024-08-30T16:03:28.408 INFO:teuthology.orchestra.run.smithi050.stdout: "last_up_change": "2024-08-30T15:59:19.280372+0000", 2024-08-30T16:03:28.408 INFO:teuthology.orchestra.run.smithi050.stdout: "last_in_change": "2024-08-30T16:03:05.847464+0000", 2024-08-30T16:03:28.408 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T16:03:28.408 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_num": 5801984, 2024-08-30T16:03:28.408 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_set": [ 2024-08-30T16:03:28.408 INFO:teuthology.orchestra.run.smithi050.stdout: "noscrub", 2024-08-30T16:03:28.408 INFO:teuthology.orchestra.run.smithi050.stdout: "pglog_hardlimit", 2024-08-30T16:03:28.408 INFO:teuthology.orchestra.run.smithi050.stdout: "purged_snapdirs", 2024-08-30T16:03:28.408 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_deletes", 2024-08-30T16:03:28.408 INFO:teuthology.orchestra.run.smithi050.stdout: "sortbitwise" 2024-08-30T16:03:28.408 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:03:28.408 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_version": 7, 2024-08-30T16:03:28.408 INFO:teuthology.orchestra.run.smithi050.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T16:03:28.409 INFO:teuthology.orchestra.run.smithi050.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T16:03:28.409 INFO:teuthology.orchestra.run.smithi050.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T16:03:28.409 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_snapshot": "", 2024-08-30T16:03:28.409 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_max": 2, 2024-08-30T16:03:28.409 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd": 8, 2024-08-30T16:03:28.409 INFO:teuthology.orchestra.run.smithi050.stdout: "require_min_compat_client": "luminous", 2024-08-30T16:03:28.409 INFO:teuthology.orchestra.run.smithi050.stdout: "min_compat_client": "jewel", 2024-08-30T16:03:28.409 INFO:teuthology.orchestra.run.smithi050.stdout: "require_osd_release": "reef", 2024-08-30T16:03:28.409 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_crimson": false, 2024-08-30T16:03:28.409 INFO:teuthology.orchestra.run.smithi050.stdout: "pools": [ 2024-08-30T16:03:28.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.409 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 1, 2024-08-30T16:03:28.409 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".mgr", 2024-08-30T16:03:28.410 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-08-30T15:59:08.355540+0000", 2024-08-30T16:03:28.410 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-08-30T16:03:28.410 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-08-30T16:03:28.410 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-08-30T16:03:28.410 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 2, 2024-08-30T16:03:28.410 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 1, 2024-08-30T16:03:28.410 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-08-30T16:03:28.410 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-08-30T16:03:28.410 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-08-30T16:03:28.410 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T16:03:28.410 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T16:03:28.410 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-08-30T16:03:28.410 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-08-30T16:03:28.410 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 1, 2024-08-30T16:03:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 1, 2024-08-30T16:03:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 1, 2024-08-30T16:03:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 1, 2024-08-30T16:03:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 1, 2024-08-30T16:03:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-08-30T16:03:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-08-30T16:03:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-08-30T16:03:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-08-30T16:03:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-08-30T16:03:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-08-30T16:03:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-08-30T16:03:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.411 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "19", 2024-08-30T16:03:28.412 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-08-30T16:03:28.412 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T16:03:28.412 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T16:03:28.412 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-08-30T16:03:28.412 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-08-30T16:03:28.412 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-08-30T16:03:28.412 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-08-30T16:03:28.412 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-08-30T16:03:28.412 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-08-30T16:03:28.412 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-08-30T16:03:28.412 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-08-30T16:03:28.412 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-08-30T16:03:28.412 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-08-30T16:03:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-08-30T16:03:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-08-30T16:03:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-08-30T16:03:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-08-30T16:03:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-08-30T16:03:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T16:03:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T16:03:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T16:03:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-08-30T16:03:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-08-30T16:03:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-08-30T16:03:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-08-30T16:03:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-08-30T16:03:28.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.414 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-08-30T16:03:28.414 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-08-30T16:03:28.414 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-08-30T16:03:28.414 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-08-30T16:03:28.414 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-08-30T16:03:28.414 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T16:03:28.414 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-08-30T16:03:28.414 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-08-30T16:03:28.414 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-08-30T16:03:28.414 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-08-30T16:03:28.414 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-08-30T16:03:28.414 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-08-30T16:03:28.414 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_max": 32, 2024-08-30T16:03:28.414 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_min": 1 2024-08-30T16:03:28.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.415 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-08-30T16:03:28.415 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": {} 2024-08-30T16:03:28.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.415 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-08-30T16:03:28.415 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 8, 2024-08-30T16:03:28.415 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 8, 2024-08-30T16:03:28.415 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.25, 2024-08-30T16:03:28.415 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-08-30T16:03:28.415 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-08-30T16:03:28.415 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-08-30T16:03:28.415 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-08-30T16:03:28.415 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-08-30T16:03:28.416 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.416 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-08-30T16:03:28.416 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "rbd", 2024-08-30T16:03:28.416 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-08-30T15:59:21.696796+0000", 2024-08-30T16:03:28.416 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-08-30T16:03:28.416 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-08-30T16:03:28.416 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-08-30T16:03:28.416 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 2, 2024-08-30T16:03:28.416 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 1, 2024-08-30T16:03:28.416 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-08-30T16:03:28.416 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-08-30T16:03:28.417 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-08-30T16:03:28.417 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T16:03:28.417 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T16:03:28.417 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-08-30T16:03:28.417 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-08-30T16:03:28.417 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 8, 2024-08-30T16:03:28.417 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 8, 2024-08-30T16:03:28.417 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 8, 2024-08-30T16:03:28.417 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 8, 2024-08-30T16:03:28.417 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 8, 2024-08-30T16:03:28.417 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-08-30T16:03:28.417 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-08-30T16:03:28.417 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-08-30T16:03:28.418 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-08-30T16:03:28.418 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-08-30T16:03:28.418 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-08-30T16:03:28.418 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-08-30T16:03:28.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.418 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "28", 2024-08-30T16:03:28.418 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-08-30T16:03:28.418 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T16:03:28.418 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T16:03:28.418 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-08-30T16:03:28.418 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-08-30T16:03:28.418 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 3, 2024-08-30T16:03:28.418 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 28, 2024-08-30T16:03:28.419 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-08-30T16:03:28.419 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-08-30T16:03:28.419 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-08-30T16:03:28.419 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-08-30T16:03:28.419 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-08-30T16:03:28.419 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-08-30T16:03:28.419 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-08-30T16:03:28.419 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-08-30T16:03:28.419 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-08-30T16:03:28.419 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-08-30T16:03:28.419 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-08-30T16:03:28.419 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T16:03:28.419 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T16:03:28.419 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T16:03:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-08-30T16:03:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-08-30T16:03:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-08-30T16:03:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-08-30T16:03:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-08-30T16:03:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-08-30T16:03:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-08-30T16:03:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-08-30T16:03:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-08-30T16:03:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-08-30T16:03:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T16:03:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-08-30T16:03:28.420 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-08-30T16:03:28.421 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-08-30T16:03:28.421 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-08-30T16:03:28.421 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-08-30T16:03:28.421 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-08-30T16:03:28.421 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-08-30T16:03:28.421 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd": {} 2024-08-30T16:03:28.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.421 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-08-30T16:03:28.421 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 4.9699997901916504, 2024-08-30T16:03:28.421 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 4.9699997901916504, 2024-08-30T16:03:28.421 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.62999999523162842, 2024-08-30T16:03:28.421 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 3.130000114440918, 2024-08-30T16:03:28.421 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 3.130000114440918, 2024-08-30T16:03:28.422 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-08-30T16:03:28.422 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-08-30T16:03:28.422 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-08-30T16:03:28.422 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.422 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.422 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:03:28.422 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-08-30T16:03:28.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.422 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-08-30T16:03:28.422 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4", 2024-08-30T16:03:28.422 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:28.422 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:28.422 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:28.422 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:28.423 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:28.423 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:28.423 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 15, 2024-08-30T16:03:28.423 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 60, 2024-08-30T16:03:28.423 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:28.423 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:28.423 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:28.423 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.423 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.423 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6812", 2024-08-30T16:03:28.423 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:03:28.423 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.423 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.424 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:28.424 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.424 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.424 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6813", 2024-08-30T16:03:28.424 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:03:28.424 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.424 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.424 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:28.424 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.425 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.425 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6815", 2024-08-30T16:03:28.425 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:03:28.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.425 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.425 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:28.425 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.425 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.425 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6814", 2024-08-30T16:03:28.425 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:03:28.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.426 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.426 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6812/1957399494", 2024-08-30T16:03:28.426 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6813/1957399494", 2024-08-30T16:03:28.426 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6815/1957399494", 2024-08-30T16:03:28.426 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6814/1957399494", 2024-08-30T16:03:28.426 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:28.426 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:28.426 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:28.426 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.426 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-08-30T16:03:28.427 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "0fe11a1c-4326-4bfd-a2a4-bff14023f301", 2024-08-30T16:03:28.427 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:28.427 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:28.427 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:28.427 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:28.427 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:28.427 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:28.427 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 22, 2024-08-30T16:03:28.427 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 22, 2024-08-30T16:03:28.427 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:28.427 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:28.427 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:28.427 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6808", 2024-08-30T16:03:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:03:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6809", 2024-08-30T16:03:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:03:28.428 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6811", 2024-08-30T16:03:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:03:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.430 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.430 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6810", 2024-08-30T16:03:28.430 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:03:28.430 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.430 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.430 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6808/3208178756", 2024-08-30T16:03:28.430 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6809/3208178756", 2024-08-30T16:03:28.430 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6811/3208178756", 2024-08-30T16:03:28.430 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6810/3208178756", 2024-08-30T16:03:28.430 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:28.430 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:28.430 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:28.431 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-08-30T16:03:28.431 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "a9d5354b-3600-41ed-80c5-a3439da7aca6", 2024-08-30T16:03:28.431 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:28.431 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:03:28.431 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:03:28.431 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:28.431 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:28.431 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:28.431 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 16, 2024-08-30T16:03:28.431 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:03:28.431 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:28.432 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:28.432 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:28.432 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.432 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.432 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6804", 2024-08-30T16:03:28.432 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:03:28.432 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.432 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.432 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:28.432 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.433 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.433 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6805", 2024-08-30T16:03:28.433 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:03:28.433 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.433 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.433 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:28.433 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.433 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.433 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6807", 2024-08-30T16:03:28.433 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:03:28.433 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.434 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.434 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:28.434 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.434 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.434 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6806", 2024-08-30T16:03:28.434 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:03:28.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.434 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.434 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6804/287018800", 2024-08-30T16:03:28.434 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6805/287018800", 2024-08-30T16:03:28.435 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6807/287018800", 2024-08-30T16:03:28.435 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6806/287018800", 2024-08-30T16:03:28.435 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:28.435 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:28.435 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:28.435 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-08-30T16:03:28.435 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "78bcc398-a7e4-4efc-9836-1dfb94407c9c", 2024-08-30T16:03:28.435 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:28.435 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:28.435 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:28.436 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:28.436 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:28.436 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:28.436 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:03:28.436 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 64, 2024-08-30T16:03:28.436 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:28.436 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:28.436 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:28.436 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.436 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.436 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6800", 2024-08-30T16:03:28.436 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:03:28.437 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.437 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.437 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:28.437 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.437 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.437 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6801", 2024-08-30T16:03:28.437 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:03:28.437 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.437 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.438 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:28.438 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.438 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.438 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6803", 2024-08-30T16:03:28.438 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:03:28.438 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.438 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.438 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:28.438 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.438 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6802", 2024-08-30T16:03:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:03:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6800/1140717714", 2024-08-30T16:03:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6801/1140717714", 2024-08-30T16:03:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6803/1140717714", 2024-08-30T16:03:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6802/1140717714", 2024-08-30T16:03:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-08-30T16:03:28.440 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "53a1060b-7985-4c2c-820e-64b158056113", 2024-08-30T16:03:28.440 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:28.440 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:03:28.440 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:03:28.440 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:28.440 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:28.440 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:28.440 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 20, 2024-08-30T16:03:28.440 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 24, 2024-08-30T16:03:28.440 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6805", 2024-08-30T16:03:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:03:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.441 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6806", 2024-08-30T16:03:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:03:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6808", 2024-08-30T16:03:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:03:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.442 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.443 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:28.443 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.443 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.443 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6807", 2024-08-30T16:03:28.443 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:03:28.443 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.443 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.443 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6805/1468073345", 2024-08-30T16:03:28.443 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6806/1468073345", 2024-08-30T16:03:28.443 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6808/1468073345", 2024-08-30T16:03:28.443 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6807/1468073345", 2024-08-30T16:03:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-08-30T16:03:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "853a03f7-227d-4f98-9e5c-c29896820746", 2024-08-30T16:03:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:28.444 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:28.445 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:28.445 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 13, 2024-08-30T16:03:28.445 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 64, 2024-08-30T16:03:28.445 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:28.445 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:28.445 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:28.445 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.445 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.445 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6809", 2024-08-30T16:03:28.445 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:03:28.445 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.446 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.446 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:28.446 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.446 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.446 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6810", 2024-08-30T16:03:28.446 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:03:28.446 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.446 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.446 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:28.446 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.447 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.447 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6812", 2024-08-30T16:03:28.447 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:03:28.447 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.447 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.447 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:28.447 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.447 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.447 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6811", 2024-08-30T16:03:28.447 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:03:28.447 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.448 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.448 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6809/3876025616", 2024-08-30T16:03:28.448 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6810/3876025616", 2024-08-30T16:03:28.448 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6812/3876025616", 2024-08-30T16:03:28.448 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6811/3876025616", 2024-08-30T16:03:28.448 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:28.448 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:28.448 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:28.448 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.449 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-08-30T16:03:28.449 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "bc978060-bf28-4027-8bba-7d6d8659bfc5", 2024-08-30T16:03:28.449 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:28.449 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:03:28.449 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:03:28.449 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:28.449 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:28.449 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:28.449 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:03:28.449 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:03:28.449 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:28.449 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:28.449 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:28.450 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.450 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.450 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6801", 2024-08-30T16:03:28.450 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:03:28.450 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.450 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.450 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:28.450 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.450 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.450 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6802", 2024-08-30T16:03:28.451 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:03:28.451 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.451 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.451 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:28.451 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.451 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.451 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6804", 2024-08-30T16:03:28.451 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:03:28.451 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.452 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.452 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:28.452 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.452 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.452 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6803", 2024-08-30T16:03:28.452 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:03:28.452 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.452 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.452 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6801/128519122", 2024-08-30T16:03:28.453 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6802/128519122", 2024-08-30T16:03:28.453 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6804/128519122", 2024-08-30T16:03:28.453 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6803/128519122", 2024-08-30T16:03:28.453 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:28.453 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:28.453 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:28.453 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.453 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-08-30T16:03:28.453 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "45df5c6f-b124-46d1-9efb-62b78bc584ed", 2024-08-30T16:03:28.453 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:28.453 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:28.453 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:28.454 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:28.454 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:28.454 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:28.454 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:03:28.454 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 40, 2024-08-30T16:03:28.454 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:28.454 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:28.454 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:28.454 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.454 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.454 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6813", 2024-08-30T16:03:28.454 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:03:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6814", 2024-08-30T16:03:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:03:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:28.455 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.456 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.456 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6816", 2024-08-30T16:03:28.456 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:03:28.456 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.456 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.456 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:28.456 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:28.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.456 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:28.456 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6815", 2024-08-30T16:03:28.456 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:03:28.457 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.457 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.457 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6813/68721277", 2024-08-30T16:03:28.457 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6814/68721277", 2024-08-30T16:03:28.457 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6816/68721277", 2024-08-30T16:03:28.457 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6815/68721277", 2024-08-30T16:03:28.457 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:28.457 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:28.458 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:28.458 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:28.458 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.458 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:03:28.458 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_xinfo": [ 2024-08-30T16:03:28.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.458 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-08-30T16:03:28.458 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:28.458 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:28.458 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:28.458 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:28.458 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:28.458 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.911895+0000", 2024-08-30T16:03:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-08-30T16:03:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:04.786543+0000", 2024-08-30T16:03:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-08-30T16:03:28.460 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:28.460 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:28.460 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:28.460 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:28.460 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:03:28.460 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:04.197952+0000", 2024-08-30T16:03:28.460 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:28.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.460 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-08-30T16:03:28.460 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:28.460 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:28.460 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.680847+0000", 2024-08-30T16:03:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-08-30T16:03:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:03:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:01.937600+0000", 2024-08-30T16:03:28.461 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:28.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.462 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-08-30T16:03:28.462 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:28.462 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:28.462 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:28.462 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:28.462 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:28.462 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:02.082761+0000", 2024-08-30T16:03:28.462 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:28.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.462 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-08-30T16:03:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:03:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.401025+0000", 2024-08-30T16:03:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-08-30T16:03:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:28.463 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:28.464 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:28.464 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.129371+0000", 2024-08-30T16:03:28.464 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:28.464 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.464 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:03:28.464 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap": [], 2024-08-30T16:03:28.464 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_items": [], 2024-08-30T16:03:28.464 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_primaries": [], 2024-08-30T16:03:28.464 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_temp": [], 2024-08-30T16:03:28.464 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_temp": [], 2024-08-30T16:03:28.464 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklist": {}, 2024-08-30T16:03:28.464 INFO:teuthology.orchestra.run.smithi050.stdout: "range_blocklist": {}, 2024-08-30T16:03:28.464 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profiles": { 2024-08-30T16:03:28.465 INFO:teuthology.orchestra.run.smithi050.stdout: "default": { 2024-08-30T16:03:28.465 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-08-30T16:03:28.465 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-08-30T16:03:28.465 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-08-30T16:03:28.465 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-08-30T16:03:28.465 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van" 2024-08-30T16:03:28.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:28.465 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps_queue": [], 2024-08-30T16:03:28.465 INFO:teuthology.orchestra.run.smithi050.stdout: "new_removed_snaps": [], 2024-08-30T16:03:28.465 INFO:teuthology.orchestra.run.smithi050.stdout: "new_purged_snaps": [], 2024-08-30T16:03:28.465 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_node_flags": {}, 2024-08-30T16:03:28.465 INFO:teuthology.orchestra.run.smithi050.stdout: "device_class_flags": {}, 2024-08-30T16:03:28.466 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": { 2024-08-30T16:03:28.466 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_enabled": false, 2024-08-30T16:03:28.466 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_bucket_count": 0, 2024-08-30T16:03:28.466 INFO:teuthology.orchestra.run.smithi050.stdout: "degraded_stretch_mode": 0, 2024-08-30T16:03:28.466 INFO:teuthology.orchestra.run.smithi050.stdout: "recovering_stretch_mode": 0, 2024-08-30T16:03:28.466 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_bucket": 0 2024-08-30T16:03:28.466 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:28.466 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:28.468 INFO:tasks.thrashosds.thrasher:j is {'epoch': 71, 'fsid': '9d72e2e2-0554-42af-8e62-9b1a3a804112', 'created': '2024-08-30T15:58:20.129410+0000', 'modified': '2024-08-30T16:03:25.685938+0000', 'last_up_change': '2024-08-30T15:59:19.280372+0000', 'last_in_change': '2024-08-30T16:03:05.847464+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T15:59:08.355540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-08-30T15:59:21.696796+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.96999979019165, 'score_stable': 4.96999979019165, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 3.130000114440918, 'raw_score_stable': 3.130000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 60, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6812', 'nonce': 1957399494}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6813', 'nonce': 1957399494}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 1957399494}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6814', 'nonce': 1957399494}]}, 'public_addr': '172.21.15.50:6812/1957399494', 'cluster_addr': '172.21.15.50:6813/1957399494', 'heartbeat_back_addr': '172.21.15.50:6815/1957399494', 'heartbeat_front_addr': '172.21.15.50:6814/1957399494', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0fe11a1c-4326-4bfd-a2a4-bff14023f301', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6808', 'nonce': 3208178756}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6809', 'nonce': 3208178756}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 3208178756}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6810', 'nonce': 3208178756}]}, 'public_addr': '172.21.15.50:6808/3208178756', 'cluster_addr': '172.21.15.50:6809/3208178756', 'heartbeat_back_addr': '172.21.15.50:6811/3208178756', 'heartbeat_front_addr': '172.21.15.50:6810/3208178756', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a9d5354b-3600-41ed-80c5-a3439da7aca6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6804', 'nonce': 287018800}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6805', 'nonce': 287018800}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 287018800}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6806', 'nonce': 287018800}]}, 'public_addr': '172.21.15.50:6804/287018800', 'cluster_addr': '172.21.15.50:6805/287018800', 'heartbeat_back_addr': '172.21.15.50:6807/287018800', 'heartbeat_front_addr': '172.21.15.50:6806/287018800', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '78bcc398-a7e4-4efc-9836-1dfb94407c9c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 64, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6800', 'nonce': 1140717714}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6801', 'nonce': 1140717714}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 1140717714}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6802', 'nonce': 1140717714}]}, 'public_addr': '172.21.15.50:6800/1140717714', 'cluster_addr': '172.21.15.50:6801/1140717714', 'heartbeat_back_addr': '172.21.15.50:6803/1140717714', 'heartbeat_front_addr': '172.21.15.50:6802/1140717714', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '53a1060b-7985-4c2c-820e-64b158056113', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 1468073345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6806', 'nonce': 1468073345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6808', 'nonce': 1468073345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 1468073345}]}, 'public_addr': '172.21.15.94:6805/1468073345', 'cluster_addr': '172.21.15.94:6806/1468073345', 'heartbeat_back_addr': '172.21.15.94:6808/1468073345', 'heartbeat_front_addr': '172.21.15.94:6807/1468073345', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '853a03f7-227d-4f98-9e5c-c29896820746', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 64, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 3876025616}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6810', 'nonce': 3876025616}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6812', 'nonce': 3876025616}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 3876025616}]}, 'public_addr': '172.21.15.94:6809/3876025616', 'cluster_addr': '172.21.15.94:6810/3876025616', 'heartbeat_back_addr': '172.21.15.94:6812/3876025616', 'heartbeat_front_addr': '172.21.15.94:6811/3876025616', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bc978060-bf28-4027-8bba-7d6d8659bfc5', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 128519122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6802', 'nonce': 128519122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6804', 'nonce': 128519122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 128519122}]}, 'public_addr': '172.21.15.94:6801/128519122', 'cluster_addr': '172.21.15.94:6802/128519122', 'heartbeat_back_addr': '172.21.15.94:6804/128519122', 'heartbeat_front_addr': '172.21.15.94:6803/128519122', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '45df5c6f-b124-46d1-9efb-62b78bc584ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 68721277}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6814', 'nonce': 68721277}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6816', 'nonce': 68721277}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 68721277}]}, 'public_addr': '172.21.15.94:6813/68721277', 'cluster_addr': '172.21.15.94:6814/68721277', 'heartbeat_back_addr': '172.21.15.94:6816/68721277', 'heartbeat_front_addr': '172.21.15.94:6815/68721277', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.911895+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:04.786543+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:04.197952+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.680847+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:01.937600+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:02.082761+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.401025+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.129371+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T16:03:28.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:03:28.469 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:28.469 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:28.469 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:28.469 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:28.469 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:28.469 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:28.466+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:28.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:28.569 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:28.568+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:28.645 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:28.645 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:28.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:28.645 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:03:28.646 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:28.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:28.670 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:28.669+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:28.771 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:28.770+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:28.823 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:28.824 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:28.824 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:28.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:28.876 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:28.873+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:28.973 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:28.972+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:29.041 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:29.041 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:29.042 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:29.042 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:29.042 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:29.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:29.074 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:29.073+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:29.121 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:03:29.122 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:03:29.131 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-08-30T16:03:29.135 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-08-30T16:03:29.069347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":397859,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":137656,"kb_used_data":2552,"kb_used_omap":7,"kb_used_meta":135032,"kb_avail":471721544,"statfs":{"total":483183820800,"available":483042861056,"internally_reserved":0,"allocated":2613248,"data_stored":2059703,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":7967,"internal_metadata":138272993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.158548"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":60,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-30T16:03:24.749433+0000","last_change":"2024-08-30T15:59:27.646756+0000","last_active":"2024-08-30T16:03:24.749433+0000","last_peered":"2024-08-30T16:03:24.749433+0000","last_clean":"2024-08-30T16:03:24.749433+0000","last_became_active":"2024-08-30T15:59:25.342321+0000","last_became_peered":"2024-08-30T15:59:25.342321+0000","last_unstale":"2024-08-30T16:03:24.749433+0000","last_undegraded":"2024-08-30T16:03:24.749433+0000","last_fullsized":"2024-08-30T16:03:24.749433+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:53:42.927368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.116914037,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-08-30T16:03:19.707407+0000","last_change":"2024-08-30T16:02:40.613055+0000","last_active":"2024-08-30T16:03:19.707407+0000","last_peered":"2024-08-30T16:03:19.707407+0000","last_clean":"2024-08-30T16:03:19.707407+0000","last_became_active":"2024-08-30T16:02:40.576990+0000","last_became_peered":"2024-08-30T16:02:40.576990+0000","last_unstale":"2024-08-30T16:03:19.707407+0000","last_undegraded":"2024-08-30T16:03:19.707407+0000","last_fullsized":"2024-08-30T16:03:19.707407+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T01:33:27.937690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":68,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-30T16:03:24.743707+0000","last_change":"2024-08-30T16:02:40.454608+0000","last_active":"2024-08-30T16:03:24.743707+0000","last_peered":"2024-08-30T16:03:24.743707+0000","last_clean":"2024-08-30T16:03:24.743707+0000","last_became_active":"2024-08-30T16:02:40.448507+0000","last_became_peered":"2024-08-30T16:02:40.448507+0000","last_unstale":"2024-08-30T16:03:24.743707+0000","last_undegraded":"2024-08-30T16:03:24.743707+0000","last_fullsized":"2024-08-30T16:03:24.743707+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T00:30:34.065701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":74,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-08-30T16:03:20.467471+0000","last_change":"2024-08-30T16:03:08.888511+0000","last_active":"2024-08-30T16:03:20.467471+0000","last_peered":"2024-08-30T16:03:20.467471+0000","last_clean":"2024-08-30T16:03:20.467471+0000","last_became_active":"2024-08-30T16:03:08.882505+0000","last_became_peered":"2024-08-30T16:03:08.882505+0000","last_unstale":"2024-08-30T16:03:20.467471+0000","last_undegraded":"2024-08-30T16:03:20.467471+0000","last_fullsized":"2024-08-30T16:03:20.467471+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:06:24.414887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":74,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-30T16:03:24.734805+0000","last_change":"2024-08-30T16:01:01.435839+0000","last_active":"2024-08-30T16:03:24.734805+0000","last_peered":"2024-08-30T16:03:24.734805+0000","last_clean":"2024-08-30T16:03:24.734805+0000","last_became_active":"2024-08-30T16:01:00.974833+0000","last_became_peered":"2024-08-30T16:01:00.974833+0000","last_unstale":"2024-08-30T16:03:24.734805+0000","last_undegraded":"2024-08-30T16:03:24.734805+0000","last_fullsized":"2024-08-30T16:03:24.734805+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:09:38.575833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":58,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-30T16:03:23.153079+0000","last_change":"2024-08-30T15:59:27.825532+0000","last_active":"2024-08-30T16:03:23.153079+0000","last_peered":"2024-08-30T16:03:23.153079+0000","last_clean":"2024-08-30T16:03:23.153079+0000","last_became_active":"2024-08-30T15:59:25.469390+0000","last_became_peered":"2024-08-30T15:59:25.469390+0000","last_unstale":"2024-08-30T16:03:23.153079+0000","last_undegraded":"2024-08-30T16:03:23.153079+0000","last_fullsized":"2024-08-30T16:03:23.153079+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T17:46:39.688911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":61,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-30T16:03:24.728583+0000","last_change":"2024-08-30T15:59:27.583986+0000","last_active":"2024-08-30T16:03:24.728583+0000","last_peered":"2024-08-30T16:03:24.728583+0000","last_clean":"2024-08-30T16:03:24.728583+0000","last_became_active":"2024-08-30T15:59:24.528536+0000","last_became_peered":"2024-08-30T15:59:24.528536+0000","last_unstale":"2024-08-30T16:03:24.728583+0000","last_undegraded":"2024-08-30T16:03:24.728583+0000","last_fullsized":"2024-08-30T16:03:24.728583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:18:40.602710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":52,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-08-30T16:03:24.740563+0000","last_change":"2024-08-30T16:03:08.552899+0000","last_active":"2024-08-30T16:03:24.740563+0000","last_peered":"2024-08-30T16:03:24.740563+0000","last_clean":"2024-08-30T16:03:24.740563+0000","last_became_active":"2024-08-30T16:03:08.491323+0000","last_became_peered":"2024-08-30T16:03:08.491323+0000","last_unstale":"2024-08-30T16:03:24.740563+0000","last_undegraded":"2024-08-30T16:03:24.740563+0000","last_fullsized":"2024-08-30T16:03:24.740563+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:26:44.292348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":61,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-08-30T16:03:20.413589+0000","last_change":"2024-08-30T15:59:21.623783+0000","last_active":"2024-08-30T16:03:20.413589+0000","last_peered":"2024-08-30T16:03:20.413589+0000","last_clean":"2024-08-30T16:03:20.413589+0000","last_became_active":"2024-08-30T15:59:20.848924+0000","last_became_peered":"2024-08-30T15:59:20.848924+0000","last_unstale":"2024-08-30T16:03:20.413589+0000","last_undegraded":"2024-08-30T16:03:20.413589+0000","last_fullsized":"2024-08-30T16:03:20.413589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:14:52.575421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":795680,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280570,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27548,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":540672,"data_stored":648940,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":10.507,"5min":12.815,"15min":13.885},"min":{"1min":6.827,"5min":5.585,"15min":5.585},"max":{"1min":23.948,"5min":57.177999999999997,"15min":57.177999999999997},"last":10.17},{"interface":"front","average":{"1min":10.48,"5min":12.42,"15min":13.032999999999999},"min":{"1min":7.0199999999999996,"5min":5.0510000000000002,"15min":5.0510000000000002},"max":{"1min":23.323,"5min":58.024999999999999,"15min":58.024999999999999},"last":9.2010000000000005}]},{"osd":2,"last update":"Fri Aug 30 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":11.992000000000001,"5min":10.936999999999999,"15min":10.598000000000001},"min":{"1min":6.1050000000000004,"5min":6.1050000000000004,"15min":6.1050000000000004},"max":{"1min":21.949000000000002,"5min":73.399000000000001,"15min":73.399000000000001},"last":20.256},{"interface":"front","average":{"1min":13.382,"5min":14.949999999999999,"15min":15.035},"min":{"1min":9.1219999999999999,"5min":6.9139999999999997,"15min":6.9139999999999997},"max":{"1min":19.760000000000002,"5min":75.120000000000005,"15min":75.120000000000005},"last":16.616}]},{"osd":3,"last update":"Fri Aug 30 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":14.696999999999999,"5min":16.510000000000002,"15min":17.279},"min":{"1min":8.0280000000000005,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":20.475000000000001,"5min":43.067999999999998,"15min":43.067999999999998},"last":23.472999999999999},{"interface":"front","average":{"1min":12.359999999999999,"5min":14.872,"15min":14.909000000000001},"min":{"1min":7.6920000000000002,"5min":7.6619999999999999,"15min":7.6619999999999999},"max":{"1min":20.210000000000001,"5min":104.38200000000001,"15min":104.38200000000001},"last":22.562000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":13.632999999999999,"5min":14.609999999999999,"15min":14.722},"min":{"1min":9.3079999999999998,"5min":7.6950000000000003,"15min":7.6950000000000003},"max":{"1min":28.326000000000001,"5min":30.754999999999999,"15min":30.754999999999999},"last":13.922000000000001},{"interface":"front","average":{"1min":12.863,"5min":13.103999999999999,"15min":13.077999999999999},"min":{"1min":7.2610000000000001,"5min":7.0039999999999996,"15min":7.0039999999999996},"max":{"1min":22.158000000000001,"5min":29.818999999999999,"15min":29.818999999999999},"last":14.289999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":20.231000000000002,"5min":16.763999999999999,"15min":16.864000000000001},"min":{"1min":9.3659999999999997,"5min":8.1950000000000003,"15min":8.1950000000000003},"max":{"1min":55.784999999999997,"5min":55.784999999999997,"15min":55.784999999999997},"last":17.474},{"interface":"front","average":{"1min":18.181999999999999,"5min":19.015999999999998,"15min":19.529},"min":{"1min":9.5800000000000001,"5min":9.5800000000000001,"15min":9.5800000000000001},"max":{"1min":54.052,"5min":54.052,"15min":54.052},"last":11.208}]},{"osd":6,"last update":"Fri Aug 30 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":14.589,"5min":17.021000000000001,"15min":17.518999999999998},"min":{"1min":9.7769999999999992,"5min":9.1929999999999996,"15min":9.1929999999999996},"max":{"1min":29.425999999999998,"5min":58.201999999999998,"15min":58.201999999999998},"last":12.702999999999999},{"interface":"front","average":{"1min":16.152000000000001,"5min":16.908999999999999,"15min":17.055},"min":{"1min":9.3879999999999999,"5min":8.8390000000000004,"15min":8.8390000000000004},"max":{"1min":25.949000000000002,"5min":37.253,"15min":37.253},"last":21.402000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:02:28 2024","interfaces":[{"interface":"back","average":{"1min":15.124000000000001,"5min":18.698,"15min":19.745999999999999},"min":{"1min":8.2050000000000001,"5min":8.2050000000000001,"15min":8.2050000000000001},"max":{"1min":20.440000000000001,"5min":40.990000000000002,"15min":40.990000000000002},"last":18.971},{"interface":"front","average":{"1min":16.059999999999999,"5min":18.408999999999999,"15min":19.158000000000001},"min":{"1min":11.728,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":31.052,"5min":59.079000000000001,"15min":59.079000000000001},"last":13.276}]}]},{"osd":7,"up_from":21,"seq":90194313274,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27520,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344320,"statfs":{"total":96636764160,"available":96608583680,"internally_reserved":0,"allocated":512000,"data_stored":255371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":8.4659999999999993,"5min":9.2919999999999998,"15min":9.0730000000000004},"min":{"1min":4.8079999999999998,"5min":4.7169999999999996,"15min":4.7169999999999996},"max":{"1min":16.690000000000001,"5min":30.600999999999999,"15min":30.600999999999999},"last":5.3170000000000002},{"interface":"front","average":{"1min":10.06,"5min":9.4169999999999998,"15min":9.2210000000000001},"min":{"1min":4.9779999999999998,"5min":4.5430000000000001,"15min":4.5430000000000001},"max":{"1min":26.928000000000001,"5min":28.077000000000002,"15min":28.077000000000002},"last":8.1519999999999992}]},{"osd":1,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":11.928000000000001,"5min":12.59,"15min":11.42},"min":{"1min":5.7270000000000003,"5min":5.7270000000000003,"15min":5.7270000000000003},"max":{"1min":23.419,"5min":83.581000000000003,"15min":83.581000000000003},"last":13.987},{"interface":"front","average":{"1min":11.988,"5min":12.587999999999999,"15min":12.467000000000001},"min":{"1min":6.3719999999999999,"5min":6.3719999999999999,"15min":6.3719999999999999},"max":{"1min":27.940000000000001,"5min":81.722999999999999,"15min":81.722999999999999},"last":10.052}]},{"osd":2,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.329000000000001,"5min":13.84,"15min":13.672000000000001},"min":{"1min":6.6559999999999997,"5min":6.5970000000000004,"15min":6.5970000000000004},"max":{"1min":42.506,"5min":42.506,"15min":42.506},"last":8.6869999999999994},{"interface":"front","average":{"1min":15.164,"5min":15.433,"15min":16.823},"min":{"1min":7.673,"5min":6.9800000000000004,"15min":6.9800000000000004},"max":{"1min":43.484999999999999,"5min":43.484999999999999,"15min":43.484999999999999},"last":16.706}]},{"osd":3,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":15.202999999999999,"5min":16.597999999999999,"15min":16.596},"min":{"1min":10.531000000000001,"5min":8.8420000000000005,"15min":8.8420000000000005},"max":{"1min":35.677999999999997,"5min":65.739999999999995,"15min":65.739999999999995},"last":18.071000000000002},{"interface":"front","average":{"1min":11.984999999999999,"5min":13.198,"15min":12.842000000000001},"min":{"1min":6.2119999999999997,"5min":6.2119999999999997,"15min":6.2119999999999997},"max":{"1min":36.258000000000003,"5min":64.980000000000004,"15min":64.980000000000004},"last":11.443}]},{"osd":4,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":14.795,"5min":13.742000000000001,"15min":12.800000000000001},"min":{"1min":8.3670000000000009,"5min":7.0339999999999998,"15min":7.0339999999999998},"max":{"1min":24.486999999999998,"5min":37.746000000000002,"15min":37.746000000000002},"last":7.8760000000000003},{"interface":"front","average":{"1min":14.532999999999999,"5min":16.396999999999998,"15min":17.324999999999999},"min":{"1min":10.377000000000001,"5min":9.2070000000000007,"15min":9.2070000000000007},"max":{"1min":22.492000000000001,"5min":37.393000000000001,"15min":37.393000000000001},"last":19.137}]},{"osd":5,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":12.417999999999999,"5min":11.829000000000001,"15min":11.496},"min":{"1min":7.6440000000000001,"5min":6.2309999999999999,"15min":6.2309999999999999},"max":{"1min":23.266999999999999,"5min":29.754999999999999,"15min":29.754999999999999},"last":14.565},{"interface":"front","average":{"1min":15.071,"5min":16.338999999999999,"15min":16.789999999999999},"min":{"1min":8.407,"5min":8.407,"15min":8.407},"max":{"1min":24.846,"5min":38.618000000000002,"15min":38.618000000000002},"last":12.724}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.326000000000001,"5min":13.771000000000001,"15min":13.208},"min":{"1min":8.0060000000000002,"5min":6.3230000000000004,"15min":6.3230000000000004},"max":{"1min":19.231000000000002,"5min":42.829999999999998,"15min":42.829999999999998},"last":15.101000000000001},{"interface":"front","average":{"1min":15.842000000000001,"5min":17.606000000000002,"15min":18.088999999999999},"min":{"1min":11.712,"5min":10.749000000000001,"15min":10.749000000000001},"max":{"1min":22.695,"5min":44.491999999999997,"15min":44.491999999999997},"last":15.974}]}]},{"osd":6,"up_from":0,"seq":90194313273,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":21,"seq":90194313273,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27524,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344316,"statfs":{"total":96636764160,"available":96608579584,"internally_reserved":0,"allocated":516096,"data_stored":255390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":10.545,"5min":10.287000000000001,"15min":11.180999999999999},"min":{"1min":5.0599999999999996,"5min":4.29,"15min":4.29},"max":{"1min":27.635000000000002,"5min":50.500999999999998,"15min":50.500999999999998},"last":7.2489999999999997},{"interface":"front","average":{"1min":11.278,"5min":12.206,"15min":14.045999999999999},"min":{"1min":5.9470000000000001,"5min":5.1360000000000001,"15min":5.1360000000000001},"max":{"1min":36.720999999999997,"5min":51.965000000000003,"15min":51.965000000000003},"last":9.0250000000000004}]},{"osd":1,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":14.144,"5min":15.917,"15min":18.513999999999999},"min":{"1min":8.2029999999999994,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":31.433,"5min":137.56700000000001,"15min":137.56700000000001},"last":10.166},{"interface":"front","average":{"1min":13.198,"5min":14.635,"15min":16.535},"min":{"1min":6.4779999999999998,"5min":5.0060000000000002,"15min":5.0060000000000002},"max":{"1min":24.753,"5min":136.321,"15min":136.321},"last":8.3170000000000002}]},{"osd":2,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":12.728999999999999,"5min":14.698,"15min":16.170000000000002},"min":{"1min":7.2480000000000002,"5min":7.2480000000000002,"15min":7.2480000000000002},"max":{"1min":26.210000000000001,"5min":73.831000000000003,"15min":73.831000000000003},"last":11.112},{"interface":"front","average":{"1min":10.778,"5min":13.547000000000001,"15min":15.545},"min":{"1min":6.7050000000000001,"5min":6.7050000000000001,"15min":6.7050000000000001},"max":{"1min":19.884,"5min":71.831999999999994,"15min":71.831999999999994},"last":12.362}]},{"osd":4,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":14.756,"5min":21.741,"15min":28.48},"min":{"1min":7.7949999999999999,"5min":7.7949999999999999,"15min":7.7949999999999999},"max":{"1min":32.465000000000003,"5min":342.34500000000003,"15min":342.34500000000003},"last":14.872999999999999},{"interface":"front","average":{"1min":16.163,"5min":17.98,"15min":20.091000000000001},"min":{"1min":10.538,"5min":8.0299999999999994,"15min":8.0299999999999994},"max":{"1min":29.649999999999999,"5min":127.857,"15min":127.857},"last":14.352}]},{"osd":5,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.073,"5min":15.792999999999999,"15min":16},"min":{"1min":7.5250000000000004,"5min":7.2039999999999997,"15min":7.2039999999999997},"max":{"1min":55.005000000000003,"5min":58.715000000000003,"15min":58.715000000000003},"last":13.471},{"interface":"front","average":{"1min":19.274000000000001,"5min":18.774000000000001,"15min":19.256},"min":{"1min":7.5869999999999997,"5min":7.2370000000000001,"15min":7.2370000000000001},"max":{"1min":54.058,"5min":58.212000000000003,"15min":58.212000000000003},"last":16.640000000000001}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.295000000000002,"5min":23.492000000000001,"15min":30.053999999999998},"min":{"1min":11.096,"5min":8.6630000000000003,"15min":8.6630000000000003},"max":{"1min":25.298999999999999,"5min":388.96100000000001,"15min":388.96100000000001},"last":17.300999999999998},{"interface":"front","average":{"1min":13.961,"5min":24.18,"15min":32.835999999999999},"min":{"1min":9.4079999999999995,"5min":8.5519999999999996,"15min":8.5519999999999996},"max":{"1min":20.439,"5min":448.69200000000001,"15min":448.69200000000001},"last":15.858000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":17.707999999999998,"5min":21.858000000000001,"15min":25.350999999999999},"min":{"1min":10.74,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":33.82,"5min":177.596,"15min":177.596},"last":18.199000000000002},{"interface":"front","average":{"1min":17.922000000000001,"5min":22.812000000000001,"15min":26.838000000000001},"min":{"1min":11.428000000000001,"5min":9.6270000000000007,"15min":9.6270000000000007},"max":{"1min":28.056999999999999,"5min":231.03800000000001,"15min":231.03800000000001},"last":18.873000000000001}]}]},{"osd":4,"up_from":0,"seq":85899345977,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":68719476791,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":15,"seq":64424509494,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27512,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344328,"statfs":{"total":96636764160,"available":96608591872,"internally_reserved":0,"allocated":503808,"data_stored":251081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":10.815,"5min":10.121,"15min":9.8149999999999995},"min":{"1min":5.0019999999999998,"5min":5.0019999999999998,"15min":5.0019999999999998},"max":{"1min":34.015000000000001,"5min":34.015000000000001,"15min":34.015000000000001},"last":7.9020000000000001},{"interface":"front","average":{"1min":10.691000000000001,"5min":11.101000000000001,"15min":11.269},"min":{"1min":4.7210000000000001,"5min":4.7210000000000001,"15min":4.7210000000000001},"max":{"1min":36.636000000000003,"5min":36.636000000000003,"15min":36.636000000000003},"last":29.138000000000002}]},{"osd":2,"last update":"Fri Aug 30 16:03:13 2024","interfaces":[{"interface":"back","average":{"1min":11.630000000000001,"5min":11.348000000000001,"15min":11.256},"min":{"1min":6.1109999999999998,"5min":3.3370000000000002,"15min":3.3370000000000002},"max":{"1min":21.43,"5min":46.768999999999998,"15min":46.768999999999998},"last":30.539999999999999},{"interface":"front","average":{"1min":10.179,"5min":13.541,"15min":14.135},"min":{"1min":7.5209999999999999,"5min":5.702,"15min":5.702},"max":{"1min":19.795000000000002,"5min":43.921999999999997,"15min":43.921999999999997},"last":25.428000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":11.773999999999999,"5min":20.818999999999999,"15min":23.815999999999999},"min":{"1min":6.2839999999999998,"5min":6.2839999999999998,"15min":6.2839999999999998},"max":{"1min":26.286000000000001,"5min":200.155,"15min":200.155},"last":36.070999999999998},{"interface":"front","average":{"1min":14.169,"5min":25.681999999999999,"15min":29.359000000000002},"min":{"1min":7.0940000000000003,"5min":7.0940000000000003,"15min":7.0940000000000003},"max":{"1min":28.536000000000001,"5min":267.03100000000001,"15min":267.03100000000001},"last":37.436}]},{"osd":4,"last update":"Fri Aug 30 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":14.153,"5min":15.212999999999999,"15min":15.74},"min":{"1min":7.3209999999999997,"5min":6.6429999999999998,"15min":6.6429999999999998},"max":{"1min":27.927,"5min":56.200000000000003,"15min":56.200000000000003},"last":20.492000000000001},{"interface":"front","average":{"1min":14.789,"5min":15.688000000000001,"15min":15.723000000000001},"min":{"1min":9.9090000000000007,"5min":8.609,"15min":8.609},"max":{"1min":28.346,"5min":48.534999999999997,"15min":48.534999999999997},"last":46.875999999999998}]},{"osd":5,"last update":"Fri Aug 30 16:03:13 2024","interfaces":[{"interface":"back","average":{"1min":15.835000000000001,"5min":15.973000000000001,"15min":16.18},"min":{"1min":11.196,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":35.521000000000001,"5min":35.521000000000001,"15min":35.521000000000001},"last":26.238},{"interface":"front","average":{"1min":17.591000000000001,"5min":17.614000000000001,"15min":17.827999999999999},"min":{"1min":9.2880000000000003,"5min":4.75,"15min":4.75},"max":{"1min":36.832000000000001,"5min":49.396000000000001,"15min":49.396000000000001},"last":39.359000000000002}]},{"osd":6,"last update":"Fri Aug 30 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":13.468,"5min":14.417,"15min":14.428000000000001},"min":{"1min":7.7850000000000001,"5min":7.6449999999999996,"15min":7.6449999999999996},"max":{"1min":59.328000000000003,"5min":59.328000000000003,"15min":59.328000000000003},"last":21.347000000000001},{"interface":"front","average":{"1min":16.844999999999999,"5min":18.213000000000001,"15min":18.495000000000001},"min":{"1min":10.005000000000001,"5min":10.005000000000001,"15min":10.005000000000001},"max":{"1min":58.567,"5min":58.567,"15min":58.567},"last":27.573}]},{"osd":7,"last update":"Fri Aug 30 16:02:31 2024","interfaces":[{"interface":"back","average":{"1min":17.196000000000002,"5min":20.622,"15min":21.584},"min":{"1min":10.632999999999999,"5min":9.1180000000000003,"15min":9.1180000000000003},"max":{"1min":33.680999999999997,"5min":59.515999999999998,"15min":59.515999999999998},"last":42.585000000000001},{"interface":"front","average":{"1min":14.763999999999999,"5min":15.246,"15min":15.597},"min":{"1min":7.968,"5min":7.1619999999999999,"15min":7.1619999999999999},"max":{"1min":35.167000000000002,"5min":58.807000000000002,"15min":58.807000000000002},"last":9.3550000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27552,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":540672,"data_stored":648921,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:03:17 2024","interfaces":[{"interface":"back","average":{"1min":10.364000000000001,"5min":11.343,"15min":13.547000000000001},"min":{"1min":4.1299999999999999,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":53.558999999999997,"5min":111.313,"15min":111.313},"last":5.5270000000000001},{"interface":"front","average":{"1min":11.138999999999999,"5min":12.304,"15min":14.882},"min":{"1min":5.4809999999999999,"5min":4.7750000000000004,"15min":4.7750000000000004},"max":{"1min":52.691000000000003,"5min":107.937,"15min":107.937},"last":17.260000000000002}]},{"osd":1,"last update":"Fri Aug 30 16:02:29 2024","interfaces":[{"interface":"back","average":{"1min":9.2829999999999995,"5min":12.077999999999999,"15min":13.07},"min":{"1min":4.657,"5min":4.657,"15min":4.657},"max":{"1min":14.106999999999999,"5min":46.988,"15min":46.988},"last":10.42},{"interface":"front","average":{"1min":12.285,"5min":15.077,"15min":15.913},"min":{"1min":7.0780000000000003,"5min":7.0780000000000003,"15min":7.0780000000000003},"max":{"1min":18.556000000000001,"5min":47.661000000000001,"15min":47.661000000000001},"last":18.372}]},{"osd":2,"last update":"Fri Aug 30 16:03:17 2024","interfaces":[{"interface":"back","average":{"1min":16.106000000000002,"5min":17.030000000000001,"15min":18.556000000000001},"min":{"1min":9.6470000000000002,"5min":4.7220000000000004,"15min":4.7220000000000004},"max":{"1min":41.549999999999997,"5min":84.317999999999998,"15min":84.317999999999998},"last":27.039000000000001},{"interface":"front","average":{"1min":14.890000000000001,"5min":15.475,"15min":16.731000000000002},"min":{"1min":6.9530000000000003,"5min":6.0279999999999996,"15min":6.0279999999999996},"max":{"1min":43.314999999999998,"5min":99.884,"15min":99.884},"last":24.204999999999998}]},{"osd":3,"last update":"Fri Aug 30 16:02:22 2024","interfaces":[{"interface":"back","average":{"1min":14.414,"5min":17.748999999999999,"15min":17.314},"min":{"1min":8.9529999999999994,"5min":8.8049999999999997,"15min":8.8049999999999997},"max":{"1min":21.344999999999999,"5min":117.346,"15min":117.346},"last":18.994},{"interface":"front","average":{"1min":13.321999999999999,"5min":16.809000000000001,"15min":16.605},"min":{"1min":6.3600000000000003,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":19.821000000000002,"5min":119.23099999999999,"15min":119.23099999999999},"last":11.055999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:02:22 2024","interfaces":[{"interface":"back","average":{"1min":15.978,"5min":17.859999999999999,"15min":18.018000000000001},"min":{"1min":9.9420000000000002,"5min":9.9420000000000002,"15min":9.9420000000000002},"max":{"1min":32.887,"5min":38.095999999999997,"15min":38.095999999999997},"last":25.959},{"interface":"front","average":{"1min":15.465,"5min":17.143999999999998,"15min":17.417999999999999},"min":{"1min":8.4849999999999994,"5min":8.4849999999999994,"15min":8.4849999999999994},"max":{"1min":30.841000000000001,"5min":40.649000000000001,"15min":40.649000000000001},"last":11.744}]},{"osd":6,"last update":"Fri Aug 30 16:02:22 2024","interfaces":[{"interface":"back","average":{"1min":10.377000000000001,"5min":11.022,"15min":11.137},"min":{"1min":6.4089999999999998,"5min":6.367,"15min":6.367},"max":{"1min":18.053999999999998,"5min":26.344999999999999,"15min":26.344999999999999},"last":8.4879999999999995},{"interface":"front","average":{"1min":15.497,"5min":17.227,"15min":17.239000000000001},"min":{"1min":10.471,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":22.446999999999999,"5min":32.237000000000002,"15min":32.237000000000002},"last":20.425999999999998}]},{"osd":7,"last update":"Fri Aug 30 16:02:22 2024","interfaces":[{"interface":"back","average":{"1min":15.763,"5min":17.344999999999999,"15min":18.108000000000001},"min":{"1min":9.7759999999999998,"5min":9.0909999999999993,"15min":9.0909999999999993},"max":{"1min":25.5,"5min":43.604999999999997,"15min":43.604999999999997},"last":12.590999999999999},{"interface":"front","average":{"1min":12.769,"5min":16.184999999999999,"15min":18.003},"min":{"1min":7.5919999999999996,"5min":6.5570000000000004,"15min":6.5570000000000004},"max":{"1min":24.512,"5min":88.278000000000006,"15min":88.278000000000006},"last":13.725}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:03:29.136 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [4, 1] 2024-08-30T16:03:29.137 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.7', '4', '1'] 2024-08-30T16:03:29.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.7 4 1 2024-08-30T16:03:29.188 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:29.187+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:29.231 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:29.231 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:29.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:29.231 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:29.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.232 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:29.232 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:29.232 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 247.95669769899999, 2024-08-30T16:03:29.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:03:29.232 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:29.232 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:29.232 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:29.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:29.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:29.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:29.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:29.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:29.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:29.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:29.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:03:29.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:03:29.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:29.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:03:29.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:03:29.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:29.234 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:03:29.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:03:29.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:29.234 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:03:29.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:03:29.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:29.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:03:29.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:03:29.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:29.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:03:29.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:03:29.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:29.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:03:29.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:03:29.235 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:29.236 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:29.236 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:29.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.236 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:29.236 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:29.236 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 247.68377033799999, 2024-08-30T16:03:29.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:03:29.236 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:29.236 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:29.236 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:29.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:29.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:29.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:29.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:29.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:29.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:29.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:29.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:03:29.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:03:29.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:03:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:03:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:03:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:03:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:03:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:03:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:03:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:03:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:03:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:03:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:03:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:03:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:29.240 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:29.240 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:29.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:29.285 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:29.284+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:29.377 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:03:29.378 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:29.375+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:29.388 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:29.388 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:29.388 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:29.388 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:29.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:29.478 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:29.476+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:29.579 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:29.578+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:29.592 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:29.592 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:29.592 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:29.592 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:29.592 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:29.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:29.642 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:29.643 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:29.680 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:29.678+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:29.737 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:29.737 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:29.781 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:29.780+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:29.806 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:03:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 244.429392767, 2024-08-30T16:03:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:03:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:29.807 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:29.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:03:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:03:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:03:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:03:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:03:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:03:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:03:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:03:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:03:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:03:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:03:29.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:03:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:03:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:03:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:03:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:03:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:29.812 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:29.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:29.875 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:29.876 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:29.881 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:29.880+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:30.004 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:30.003+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:30.005 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:30.006 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:30.006 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:30.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:30.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '22 of 30' 2024-08-30T16:03:30.083 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:30.081+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:30.091 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:30.091 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:30.153 INFO:teuthology.orchestra.run.smithi050.stderr:set 2.7 pg_upmap mapping to [4,1] 2024-08-30T16:03:30.184 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:30.183+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:30.192 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:30.192 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:30.209 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:30.209 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:30.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:30.275 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:03:30.276 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:03:30.285 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:30.284+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:30.372 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:30.372 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:30.390 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:30.390 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:30.391 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:30.389+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:30.399 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:30.487 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:30.486+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:30.524 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:30.524 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:30.534 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:30.589 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:30.588+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:30.658 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:30.658 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:30.668 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:30.690 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:30.688+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:30.790 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:30.789+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:30.794 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:30.794 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:30.803 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:30.891 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:30.890+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:30.927 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:30.927 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:30.936 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:30.992 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:30.990+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:31.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '23 of 30' 2024-08-30T16:03:31.059 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:31.060 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:31.069 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:31.122 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:31.121+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:31.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-08-30T16:03:31.193 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:31.191+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:31.231 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:31.232 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:31.241 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:03:31.325 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:31.323+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:31.394 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:31.392+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:31.404 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:31.404 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:31.413 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-08-30T16:03:31.498 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:31.496+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:31.577 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:31.577 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:31.586 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-08-30T16:03:31.596 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:31.594+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:31.697 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:31.695+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:31.749 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:31.750 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:31.759 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:31.797 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:31.796+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:31.899 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:31.897+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:31.923 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:31.923 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:31.933 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:32.000 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:31.998+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:32.057 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:32.057 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:32.065 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:32.101 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:32.099+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:32.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '24 of 30' 2024-08-30T16:03:32.175 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-08-30T16:03:32.230 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:32.229+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:32.234 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:32.234 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:32.245 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:32.311 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:32.306+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:32.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:03:32.386 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:32.386 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:32.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:32.403 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:32.402+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:32.504 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:32.503+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:32.589 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:32.590 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:32.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:32.605 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:32.603+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:32.663 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:32.663 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:32.706 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:32.704+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:32.739 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:32.739 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:32.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:32.739 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:03:32.739 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:32.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:32.762 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:32.762 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:32.857 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:32.858 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:32.858 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:32.856+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:32.910 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:32.908+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:32.910 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:32.911 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:32.911 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:32.911 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:32.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:32.972 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:32.972 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:33.010 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:33.009+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:33.063 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:33.092 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:33.093 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:33.093 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:33.093 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:33.093 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:33.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:33.152 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:33.153 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:33.153 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:33.151+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:33.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '25 of 30' 2024-08-30T16:03:33.231 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:33.230+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:33.236 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-08-30T16:03:33.236 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T16:03:33.282 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:33.282 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:33.282 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:33.282 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:33.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.282 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:33.282 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:33.282 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 252.00730718299999, 2024-08-30T16:03:33.282 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:03:33.282 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:33.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:03:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:03:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:03:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:03:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:03:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:03:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:03:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:03:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:03:33.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:03:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:03:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:03:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:03:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:03:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:33.286 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 251.73437982199999, 2024-08-30T16:03:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:03:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:33.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:03:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:03:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:03:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:03:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:03:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:03:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:03:33.289 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:03:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:03:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:03:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:03:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:03:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.291 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:33.291 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:03:33.291 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:03:33.291 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:33.291 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:33.291 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:33.291 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:33.291 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:33.291 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:33.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:33.320 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:33.321 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:33.343 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:33.336+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:33.416 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:33.415+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:33.459 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:33.459 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:33.459 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:33.459 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:33.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:33.516 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:33.514+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:33.617 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:33.616+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:33.636 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:33.636 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:33.636 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:33.636 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:33.636 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:33.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:33.730 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:33.729+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:33.819 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:33.818+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:33.820 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:33.820 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:33.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:33.820 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:33.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.820 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:03:33.820 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:33.820 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 248.44276297499999, 2024-08-30T16:03:33.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:03:33.821 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:33.821 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:33.821 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:33.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:33.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:33.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:33.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:33.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:33.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:33.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:33.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:03:33.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:03:33.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:33.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:03:33.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:03:33.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:33.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:03:33.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:03:33.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:33.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:03:33.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:03:33.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:33.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:03:33.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:03:33.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:33.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:03:33.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:03:33.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:33.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:03:33.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:03:33.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:33.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:33.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:33.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:03:33.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:03:33.825 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:33.825 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:33.825 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:33.825 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:33.825 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:33.825 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:33.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:33.920 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:33.919+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:33.999 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:33.999 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:33.999 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:33.999 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:34.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:34.021 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:34.019+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:34.122 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:34.121+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:34.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-08-30T16:03:34.191 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:34.191 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:34.191 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:34.191 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:34.191 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:34.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:34.243 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:34.242+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:34.324 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:34.323+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:34.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '26 of 30' 2024-08-30T16:03:34.409 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:34.409 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:34.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:34.409 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:34.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.409 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:34.409 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:34.409 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 249.95387377, 2024-08-30T16:03:34.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:03:34.409 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:34.410 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:34.410 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:34.410 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:34.410 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:34.410 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:34.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.410 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:34.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:34.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:34.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:34.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:34.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:34.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:03:34.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:03:34.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:34.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:03:34.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:03:34.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:34.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:03:34.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:03:34.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:34.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:03:34.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:03:34.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:34.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:03:34.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:03:34.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:03:34.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:03:34.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:03:34.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:34.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:03:34.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:03:34.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:34.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:03:34.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:03:34.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:03:34.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:03:34.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:03:34.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:34.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:03:34.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:03:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:03:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:03:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:03:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:03:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:34.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:03:34.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:03:34.416 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.416 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:34.416 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.416 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:03:34.416 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:34.416 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 248.83800743500001, 2024-08-30T16:03:34.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:03:34.416 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:34.416 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:34.417 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:34.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:34.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:34.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:34.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:34.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:34.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:03:34.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:34.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:34.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:03:34.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:03:34.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:34.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:03:34.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:03:34.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:34.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:03:34.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:03:34.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:34.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:03:34.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:03:34.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:34.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:03:34.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:03:34.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:34.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:03:34.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:03:34.420 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.420 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:34.420 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.420 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:34.420 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 248.77841593700001, 2024-08-30T16:03:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:03:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:03:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:34.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:34.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:34.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:03:34.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:03:34.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:34.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:03:34.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:03:34.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:34.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:03:34.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:03:34.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:34.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:03:34.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:03:34.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:03:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:03:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:03:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:03:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.425 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:03:34.425 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:34.425 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 248.75171725000001, 2024-08-30T16:03:34.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:03:34.425 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:34.425 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:34.425 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:34.425 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:34.425 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:34.425 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:34.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.425 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:34.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:34.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:34.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:34.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:34.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:34.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:03:34.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:34.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:34.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:03:34.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:03:34.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:34.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:03:34.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:03:34.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:34.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:03:34.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:03:34.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:34.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:03:34.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:03:34.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:34.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:03:34.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:03:34.429 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.430 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:34.430 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.430 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:34.430 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:34.430 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 247.884100722, 2024-08-30T16:03:34.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:03:34.430 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:34.430 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:34.430 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:34.430 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:34.431 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:34.431 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:34.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.431 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:34.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:34.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:34.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:34.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:34.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:34.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:03:34.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:34.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:34.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:03:34.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:03:34.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:34.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:03:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:03:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:03:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:03:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:03:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:03:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:34.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:03:34.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:03:34.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:34.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:03:34.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:03:34.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:34.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:03:34.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:03:34.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:03:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:03:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:03:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:03:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:03:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:03:34.435 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.436 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:34.436 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.436 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:03:34.436 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:34.436 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 247.72485364299999, 2024-08-30T16:03:34.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:03:34.436 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:34.436 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:34.436 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:34.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:03:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:03:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:03:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:03:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:03:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:03:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:03:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:03:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:03:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:03:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:03:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:03:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:03:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:03:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 247.44867957700001, 2024-08-30T16:03:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:03:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:03:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:03:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:03:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:03:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:03:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:03:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:03:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:03:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:03:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:03:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:03:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 246.790387998, 2024-08-30T16:03:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:03:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:03:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:03:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:03:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:03:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:03:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:03:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:03:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:03:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:03:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:03:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:03:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:03:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:03:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:03:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:03:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:03:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:03:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:03:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:03:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:03:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:03:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:03:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:03:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 246.69447471399999, 2024-08-30T16:03:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:03:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:34.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:03:34.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:34.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:34.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:03:34.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:03:34.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:34.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:03:34.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:03:34.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:34.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:03:34.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:03:34.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:34.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:03:34.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:03:34.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:34.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:03:34.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:03:34.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:34.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:03:34.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:03:34.454 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.454 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:34.454 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.454 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:34.454 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:34.454 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 187.44311420299999, 2024-08-30T16:03:34.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:03:34.455 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:34.455 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:34.455 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:34.455 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:34.455 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:34.455 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:34.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.455 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:34.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:34.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:34.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:34.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:34.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:34.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:03:34.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:03:34.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:34.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:03:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:03:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:03:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:03:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:03:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:03:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:34.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:03:34.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:03:34.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:34.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:03:34.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:03:34.458 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.458 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:34.458 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.458 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:03:34.459 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:34.459 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 127.434009516, 2024-08-30T16:03:34.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:03:34.459 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:34.459 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:34.459 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:34.459 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:34.459 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:34.459 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:34.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.459 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:34.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:34.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:34.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:34.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:34.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:34.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:34.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:03:34.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:03:34.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:34.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:03:34.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:03:34.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:34.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:03:34.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:03:34.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:34.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:03:34.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:03:34.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:34.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:03:34.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:03:34.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:34.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:34.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:34.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:03:34.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:03:34.462 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.462 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:34.462 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.462 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:34.463 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:34.463 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:34.464 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:34.465 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:34.463+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:34.565 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:34.564+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:34.600 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:34.600 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:34.600 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:34.600 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:34.609 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:34.666 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:34.665+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:34.767 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:34.766+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:34.776 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:34.776 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:34.776 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:34.776 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:34.776 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:34.785 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:34.871 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:34.869+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:34.948 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:34.948 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:34.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:34.948 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:03:34.949 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:34.957 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:34.969 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:34.968+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:35.071 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:35.069+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:35.121 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:35.121 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:35.121 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:35.121 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:35.131 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:35.174 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7] out_osds: [6, 4, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:03:35.174 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:03:35.175 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-30T16:03:35.175 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-08-30T16:03:35.175 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:03:35.176 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:35.173+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:35.182 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:35.174+0000 adad640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 23096) UID: 0 2024-08-30T16:03:35.183 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:03:35.175+0000 adad640 -1 osd.6 74 *** Got signal Terminated *** 2024-08-30T16:03:35.254 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:35.255 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:35.255 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:35.255 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:35.255 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:35.264 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:35.271 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:35.270+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:35.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:03:35.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '27 of 30' 2024-08-30T16:03:35.381 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:35.379+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:35.397 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-08-30T16:03:35.474 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:35.472+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:35.484 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:35.484 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:35.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:35.485 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:35.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.485 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:35.485 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:35.485 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 264.61697576699999, 2024-08-30T16:03:35.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:03:35.485 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.485 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.485 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.485 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:03:35.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.486 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:35.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:35.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:03:35.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:03:35.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:03:35.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:03:35.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:03:35.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:03:35.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:03:35.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:03:35.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:03:35.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:03:35.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:03:35.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:03:35.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:03:35.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:03:35.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:03:35.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:03:35.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:03:35.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:03:35.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:03:35.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:03:35.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:03:35.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:03:35.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:03:35.491 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.491 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.491 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.491 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:35.491 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:35.491 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 263.700559747, 2024-08-30T16:03:35.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:03:35.491 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.491 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.491 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.492 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.492 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.492 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:03:35.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.492 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:35.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:35.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:03:35.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:35.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.493 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.493 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:03:35.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:03:35.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:03:35.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:03:35.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:03:35.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:03:35.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:03:35.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:03:35.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:03:35.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:03:35.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:03:35.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:03:35.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:03:35.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:03:35.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:03:35.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:03:35.496 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.496 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.496 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.496 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:35.497 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:35.497 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 263.69885519399998, 2024-08-30T16:03:35.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:03:35.497 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.497 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.497 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.497 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.497 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.497 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:03:35.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.497 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:35.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:35.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:03:35.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:03:35.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:03:35.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:03:35.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:03:35.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:03:35.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:03:35.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:03:35.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:03:35.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:03:35.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:03:35.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:03:35.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:03:35.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:03:35.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:03:35.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:03:35.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:03:35.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:03:35.502 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.502 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.502 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.502 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.502 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:35.502 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 263.27940138700001, 2024-08-30T16:03:35.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:03:35.502 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.502 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.502 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.503 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.503 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.503 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:03:35.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.503 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:35.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:35.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:03:35.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:03:35.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.504 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.504 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:03:35.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:03:35.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:03:35.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:03:35.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:03:35.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:03:35.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:03:35.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:03:35.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:03:35.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:03:35.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:03:35.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:03:35.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:03:35.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:03:35.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:03:35.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:03:35.507 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.507 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.507 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.507 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.508 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:35.508 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 263.26251321299998, 2024-08-30T16:03:35.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:03:35.508 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.508 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.508 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.508 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.508 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.508 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:03:35.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.508 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:35.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:35.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:03:35.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:03:35.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:03:35.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:03:35.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:03:35.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:03:35.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:03:35.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:03:35.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:03:35.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:03:35.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:03:35.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:03:35.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:03:35.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:03:35.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:03:35.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:03:35.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:03:35.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:03:35.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.513 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.513 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.513 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:35.513 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 263.24951314399999, 2024-08-30T16:03:35.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:03:35.513 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.513 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.513 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.513 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.514 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.514 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:03:35.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.514 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:35.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:35.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:03:35.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:03:35.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:03:35.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:03:35.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.515 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:03:35.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:03:35.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:03:35.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:03:35.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:03:35.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:03:35.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:03:35.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:03:35.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:03:35.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:03:35.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:03:35.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:03:35.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:03:35.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:03:35.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.518 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.518 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.518 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.518 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:35.518 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 263.24702085199999, 2024-08-30T16:03:35.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:03:35.519 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.519 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.519 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.519 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:03:35.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.519 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:35.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:35.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:03:35.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:35.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:03:35.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:03:35.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:03:35.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:03:35.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:03:35.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:03:35.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:03:35.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:03:35.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:03:35.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:03:35.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:03:35.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:03:35.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:03:35.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:03:35.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:03:35.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:03:35.523 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.523 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.523 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.524 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.524 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:35.524 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 263.24626340399999, 2024-08-30T16:03:35.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:03:35.524 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.524 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.524 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.524 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.524 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.524 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:03:35.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.524 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:35.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:35.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:03:35.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:03:35.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:03:35.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:03:35.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:03:35.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:03:35.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:03:35.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:03:35.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:03:35.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:03:35.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:03:35.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:03:35.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:03:35.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:03:35.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:03:35.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:03:35.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:03:35.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:03:35.529 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.529 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.529 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.529 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.529 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:35.529 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 263.245530954, 2024-08-30T16:03:35.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:03:35.529 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:03:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:03:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:03:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:03:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:03:35.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:03:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:03:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:03:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:03:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:03:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:03:35.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:03:35.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:03:35.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:03:35.534 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.534 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.534 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.534 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.534 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:35.535 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 263.24299215500002, 2024-08-30T16:03:35.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:03:35.535 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.535 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.535 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.535 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.535 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.535 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:03:35.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.535 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:35.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:35.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:03:35.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:35.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:03:35.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:03:35.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:03:35.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:03:35.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:03:35.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:03:35.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:03:35.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:03:35.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:03:35.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:03:35.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:03:35.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:03:35.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:03:35.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:03:35.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:03:35.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:03:35.539 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.540 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.540 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.540 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.540 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:35.540 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 263.24224415399999, 2024-08-30T16:03:35.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:03:35.540 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.540 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.540 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.540 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.540 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.540 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:03:35.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.541 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:35.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:35.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:03:35.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:03:35.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:03:35.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:03:35.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:03:35.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:03:35.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:03:35.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:03:35.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:03:35.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:03:35.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:03:35.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:03:35.545 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.545 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.545 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.545 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.546 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:35.546 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 262.470127682, 2024-08-30T16:03:35.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:03:35.546 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.546 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.546 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.546 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.546 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.546 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:03:35.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.546 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:35.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:35.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:03:35.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:03:35.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:03:35.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:03:35.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:03:35.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:03:35.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:03:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:03:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:03:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:03:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:03:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:03:35.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:03:35.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:03:35.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:03:35.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:03:35.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:03:35.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:03:35.551 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.551 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.551 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.551 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.551 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:35.551 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 262.46807053600003, 2024-08-30T16:03:35.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:03:35.551 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.551 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:03:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:03:35.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:35.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:03:35.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:03:35.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:03:35.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:03:35.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:03:35.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:03:35.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:03:35.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:03:35.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:03:35.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:03:35.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:03:35.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:03:35.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:03:35.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:03:35.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:03:35.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:03:35.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.556 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.556 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.556 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:35.557 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 262.20512603600002, 2024-08-30T16:03:35.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:03:35.557 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.557 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.557 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.557 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.557 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.557 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:03:35.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.558 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:35.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:35.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:03:35.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:03:35.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:03:35.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:03:35.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:03:35.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:03:35.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:03:35.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:03:35.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:03:35.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:03:35.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:03:35.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:03:35.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:03:35.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:03:35.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:03:35.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:03:35.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:03:35.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:03:35.562 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.562 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.563 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.563 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.563 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:35.563 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 262.20403453300003, 2024-08-30T16:03:35.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:03:35.563 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.563 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.563 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.563 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.564 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.564 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:03:35.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.564 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:35.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.564 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:35.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:03:35.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:35.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:03:35.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:03:35.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:03:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:03:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:03:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:03:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:03:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:03:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:03:35.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:03:35.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:03:35.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:03:35.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:03:35.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:03:35.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:03:35.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:03:35.568 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.568 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.568 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.568 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.568 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:35.569 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 262.20328494799998, 2024-08-30T16:03:35.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:03:35.569 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.569 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.569 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.569 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:03:35.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.569 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:35.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:35.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:03:35.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:35.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:03:35.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:03:35.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.571 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.571 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:03:35.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:03:35.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:03:35.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:03:35.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:03:35.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:03:35.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:03:35.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:03:35.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:03:35.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:03:35.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:03:35.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:03:35.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:03:35.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:03:35.574 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.574 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.574 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.574 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.574 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:35.574 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 262.16955341599999, 2024-08-30T16:03:35.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:03:35.574 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.574 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.574 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.575 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.575 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.575 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:03:35.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.575 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:35.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:35.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:03:35.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:03:35.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:03:35.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:03:35.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:03:35.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:03:35.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.577 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.577 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:03:35.577 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:03:35.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:03:35.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:03:35.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:03:35.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:03:35.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:03:35.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:03:35.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:03:35.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:03:35.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:03:35.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:03:35.579 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.580 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.580 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.580 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.580 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:35.580 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 262.16902740799998, 2024-08-30T16:03:35.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:03:35.580 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.580 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.580 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.580 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.581 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.581 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:03:35.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.581 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:35.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:35.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:03:35.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:35.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:03:35.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:03:35.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:03:35.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:03:35.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:03:35.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:03:35.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.583 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.583 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:03:35.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:03:35.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:03:35.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:03:35.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:03:35.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:03:35.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:03:35.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:03:35.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:03:35.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:03:35.585 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.585 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.585 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.585 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:35.586 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:35.586 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 262.16857184499997, 2024-08-30T16:03:35.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:03:35.586 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.586 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.586 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.586 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.586 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.586 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:03:35.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.586 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:03:35.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:03:35.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:03:35.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:03:35.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:03:35.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:03:35.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:03:35.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:03:35.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:03:35.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:03:35.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:03:35.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:03:35.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:03:35.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:03:35.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:03:35.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:03:35.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:03:35.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:03:35.591 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.591 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.591 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.591 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:03:35.591 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:35.591 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 257.498056529, 2024-08-30T16:03:35.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:03:35.591 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:35.591 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:35.592 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:35.592 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:35.592 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:35.592 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:03:35.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.592 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:35.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:35.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:35.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:35.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:35.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:35.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:35.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:03:35.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:35.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:35.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:03:35.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:03:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:03:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:03:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:03:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:03:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:03:35.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:03:35.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:35.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:03:35.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:03:35.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:35.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:03:35.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:03:35.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:35.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:03:35.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:03:35.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:35.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:35.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:35.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:03:35.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:03:35.596 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.596 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.596 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.596 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:35.596 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:35.596 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:35.597 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:35.598 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:35.598 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:35.616 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:35.615+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:35.685 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:35.685 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:35.699 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:35.697+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:35.764 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:35.764 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:35.764 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:35.773 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:35.777 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:35.777 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:35.817 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:35.815+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:35.868 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:35.869 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:35.896 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:35.896 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:35.897 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:35.897 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:35.897 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:35.900 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:35.899+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:35.906 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:35.953 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:35.953 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:36.001 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:36.000+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:36.042 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:36.043 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:36.050 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:03:36.099 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 247.71788609500001, 2024-08-30T16:03:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:03:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:03:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:36.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:03:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:03:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:03:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:03:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:03:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:03:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:03:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:03:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:03:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:03:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:03:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:03:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:36.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:03:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 127.60488997900001, 2024-08-30T16:03:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:03:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:03:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.105 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:03:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:03:36.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:03:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:03:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:03:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:03:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:03:36.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:03:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:03:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:03:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:03:36.109 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-08-30T16:03:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:03:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 7.5964706079999997, 2024-08-30T16:03:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037103309999999999, 2024-08-30T16:03:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:36.110 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 10 2024-08-30T16:03:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:03:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:03:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:36.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496549+0000", 2024-08-30T16:03:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:03:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496682+0000", 2024-08-30T16:03:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013371699999999999 2024-08-30T16:03:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496889+0000", 2024-08-30T16:03:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00020673600000000001 2024-08-30T16:03:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.497052+0000", 2024-08-30T16:03:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000162578 2024-08-30T16:03:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.498833+0000", 2024-08-30T16:03:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017810530000000001 2024-08-30T16:03:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.500282+0000", 2024-08-30T16:03:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001449437 2024-08-30T16:03:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:36.114 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:36.115 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:36.115 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:36.115 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:36.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:36.117 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:36.114+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:36.152 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:36.152 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:36.156 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:03:36.156 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:03:36.235 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:36.233+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:36.248 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:36.248 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:36.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:36.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '28 of 30' 2024-08-30T16:03:36.335 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:36.333+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:36.419 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:36.418+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:36.454 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:36.454 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:36.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:36.520 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:36.518+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:36.621 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:36.619+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:36.629 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:36.630 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:36.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:36.722 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:36.721+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:36.768 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:36.768 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:36.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:36.823 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:36.821+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:36.924 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:36.923+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:36.946 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:36.946 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:36.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:37.039 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:37.038+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:37.123 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:37.123 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:37.125 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:37.124+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:37.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:37.227 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:37.226+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:37.312 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:37.312 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:37.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:37.327 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:37.325+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:37.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '29 of 30' 2024-08-30T16:03:37.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-08-30T16:03:37.428 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:37.427+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:37.453 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:37.453 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:37.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:37.528 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:37.527+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:37.629 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:37.628+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:37.635 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:37.635 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:37.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:37.730 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:37.729+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:37.774 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:37.774 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:37.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:37.831 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:37.830+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:37.902 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:00.508 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:37.916 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:37.916 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:37.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:37.931 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:37.930+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:37.957 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:00.563 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:37.982 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:00.587 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:38.033 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:38.031+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:38.048 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:19.438 42904== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:38.050 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:19.440 42904== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:38.077 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:19.467 42904== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:38.109 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:38.109 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:38.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:38.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:03:38.168 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:38.166+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:38.234 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:38.233+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:38.251 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:38.251 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:38.260 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:38.335 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:38.334+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:38.402 INFO:tasks.mon_recovery:adding mon b back in 2024-08-30T16:03:38.402 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-30T16:03:38.402 DEBUG:teuthology.orchestra.run.smithi094:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-08-30T16:03:38.405 INFO:tasks.ceph.mon.b:Started 2024-08-30T16:03:38.405 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-08-30T16:03:38.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:03:38.425 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:38.425 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:38.435 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:38.478 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:38.475+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:38.487 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:38.487 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:38.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-08-30T16:03:38.583 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:38.583 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:38.584 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:38.581+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:38.601 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:38.602 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:38.611 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:38.640 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:38.636+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:38.675 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:38.675 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:38.736 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:38.740 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:38.737+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:38.746 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:38.763 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:38.764 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:38.852 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:38.852 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:38.853 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:38.850+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:38.893 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:38.893 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:38.903 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:38.941 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:38.939+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:38.957 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:38.973 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:03:39.043 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:39.040+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:39.076 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:39.076 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:03:39.081 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:03:39.121 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-08-30T16:03:39.135 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":46,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":34,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:03:39.137 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:39.137 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 268.27395895699999, 2024-08-30T16:03:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:03:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:03:39.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:03:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:03:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:03:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:03:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:03:39.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:03:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:03:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:03:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:03:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:03:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:03:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:03:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:03:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:03:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:03:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:03:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:03:39.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:03:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:03:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:03:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:03:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:03:39.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:03:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 267.35754293700001, 2024-08-30T16:03:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:03:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.144 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:03:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:03:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:03:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:03:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:03:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:03:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:03:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:03:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:03:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:03:39.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:03:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:03:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:03:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:03:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:03:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:03:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:03:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:03:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:39.149 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 267.35583838399998, 2024-08-30T16:03:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:03:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:03:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:39.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:39.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:03:39.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:03:39.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:03:39.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:03:39.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:03:39.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:03:39.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:03:39.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:03:39.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:03:39.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:03:39.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:03:39.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:03:39.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:03:39.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:03:39.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:03:39.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:03:39.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:03:39.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:03:39.154 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.155 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.155 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.155 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.155 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:39.155 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 266.93638457700001, 2024-08-30T16:03:39.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:03:39.155 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.155 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.155 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.156 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.156 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.156 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:03:39.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.156 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:39.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:39.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:03:39.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:03:39.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:03:39.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:03:39.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:03:39.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:03:39.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:03:39.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:03:39.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:03:39.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:03:39.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:03:39.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:03:39.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:03:39.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:03:39.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:03:39.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:03:39.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:03:39.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:03:39.160 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.160 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.160 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.160 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.160 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:39.161 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 266.91949640299998, 2024-08-30T16:03:39.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:03:39.161 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.161 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.161 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.161 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.161 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.161 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:03:39.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.161 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:39.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:39.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:03:39.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:03:39.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:03:39.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:03:39.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:03:39.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:03:39.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:03:39.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:03:39.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:03:39.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:03:39.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:03:39.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:03:39.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:03:39.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:03:39.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:03:39.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:03:39.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:03:39.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:03:39.165 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.165 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.166 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.166 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.166 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:39.166 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 266.906496334, 2024-08-30T16:03:39.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:03:39.166 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.166 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.166 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.166 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.166 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.166 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:03:39.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.167 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:39.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:39.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:03:39.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:03:39.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:03:39.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:03:39.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:03:39.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:03:39.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:03:39.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:03:39.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:03:39.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:03:39.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:03:39.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:03:39.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:03:39.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:03:39.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:03:39.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:03:39.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:03:39.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:03:39.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.171 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.171 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.171 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:39.171 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 266.904004042, 2024-08-30T16:03:39.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:03:39.172 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.172 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.172 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.172 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.172 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.172 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:03:39.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.172 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:39.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:39.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:03:39.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:39.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:03:39.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:03:39.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:03:39.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:03:39.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:03:39.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:03:39.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:03:39.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:03:39.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:03:39.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:03:39.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:03:39.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:03:39.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:03:39.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:03:39.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:03:39.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:03:39.177 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.177 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.177 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.177 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.177 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:39.177 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 266.903246594, 2024-08-30T16:03:39.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:03:39.177 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.177 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.178 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.178 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.178 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.178 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:03:39.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.178 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:39.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:03:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:03:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:03:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:03:39.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:03:39.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:03:39.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:03:39.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:03:39.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:03:39.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:03:39.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:03:39.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:03:39.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:03:39.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:03:39.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:03:39.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:03:39.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:03:39.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:03:39.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.183 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.183 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.183 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.183 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:39.183 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 266.90251414400001, 2024-08-30T16:03:39.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:03:39.183 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.183 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.184 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.184 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:03:39.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.184 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:39.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:39.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:03:39.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:39.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:03:39.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:03:39.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:03:39.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:03:39.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:03:39.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:03:39.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:03:39.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:03:39.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:03:39.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:03:39.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:03:39.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:03:39.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:03:39.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:03:39.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:03:39.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:03:39.188 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.188 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.188 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.189 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.189 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:39.189 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 266.89997534499997, 2024-08-30T16:03:39.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:03:39.189 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.189 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.189 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:03:39.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.189 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:39.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:39.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:03:39.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:39.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:03:39.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:03:39.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:03:39.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:03:39.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:03:39.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:03:39.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:03:39.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:03:39.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:03:39.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:03:39.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:03:39.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:03:39.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:03:39.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:03:39.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:03:39.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:03:39.194 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.194 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.194 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.194 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.194 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:39.194 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 266.899227344, 2024-08-30T16:03:39.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:03:39.194 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.195 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.195 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.195 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.195 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.195 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:03:39.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.195 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:39.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:39.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:03:39.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:03:39.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:03:39.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:03:39.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:03:39.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:03:39.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:03:39.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:03:39.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:03:39.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:03:39.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:03:39.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:03:39.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:03:39.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:03:39.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.199 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.200 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:39.200 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 266.127110872, 2024-08-30T16:03:39.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:03:39.200 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.200 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.200 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.200 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.200 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.200 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:03:39.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.200 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:39.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:39.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:03:39.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:03:39.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:03:39.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:03:39.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:03:39.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:03:39.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:03:39.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:03:39.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:03:39.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:03:39.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:03:39.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:03:39.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:03:39.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:03:39.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:03:39.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:03:39.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:03:39.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:03:39.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.205 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.205 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.205 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:39.205 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 266.12505372599998, 2024-08-30T16:03:39.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:03:39.205 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.205 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.206 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.206 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.206 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.206 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:03:39.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.206 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:39.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:39.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:03:39.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:39.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:03:39.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:03:39.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:03:39.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:03:39.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:03:39.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:03:39.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:03:39.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:03:39.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:03:39.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:03:39.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:03:39.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:03:39.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:03:39.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:03:39.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:03:39.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:03:39.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.210 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.211 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.211 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:39.211 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 265.86210922599997, 2024-08-30T16:03:39.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:03:39.211 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.211 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.211 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:03:39.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.212 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:39.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:39.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:03:39.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:03:39.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:03:39.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:03:39.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:03:39.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:03:39.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:03:39.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:03:39.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:03:39.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:03:39.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:03:39.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:03:39.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:03:39.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:03:39.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:03:39.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:03:39.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:03:39.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:03:39.216 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.216 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.216 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.216 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.216 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:39.217 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 265.86101772299997, 2024-08-30T16:03:39.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:03:39.217 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.217 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.217 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.217 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.217 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.217 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:03:39.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.217 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:39.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:39.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:03:39.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:39.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:03:39.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:03:39.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:03:39.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:03:39.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:03:39.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:03:39.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:03:39.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:03:39.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:03:39.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:03:39.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:03:39.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:03:39.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:03:39.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:03:39.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:03:39.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:03:39.222 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.222 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.222 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.222 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.222 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 265.86026813799998, 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:03:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:03:39.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:03:39.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:03:39.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:03:39.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:03:39.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:03:39.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:03:39.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:03:39.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:03:39.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:03:39.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:03:39.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:03:39.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:03:39.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:03:39.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:03:39.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:03:39.227 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.227 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.228 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.228 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.228 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:39.228 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 265.82653660599999, 2024-08-30T16:03:39.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:03:39.228 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.228 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.229 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.229 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.229 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.229 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:03:39.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.229 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:39.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:39.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:03:39.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:03:39.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:03:39.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:03:39.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:03:39.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:03:39.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:03:39.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:03:39.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:03:39.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:03:39.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:03:39.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:03:39.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:03:39.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:03:39.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:03:39.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:03:39.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:03:39.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 265.82601059799998, 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.234 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:03:39.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.235 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:39.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:39.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:03:39.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:39.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:03:39.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:03:39.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:03:39.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:03:39.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:03:39.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:03:39.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:03:39.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:03:39.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:03:39.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:03:39.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:03:39.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:03:39.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:03:39.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:03:39.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:03:39.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:03:39.239 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.239 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.239 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.239 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:39.239 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:39.240 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 265.82555503499998, 2024-08-30T16:03:39.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:03:39.240 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.240 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.240 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.240 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.240 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.240 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:03:39.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.240 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:39.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:39.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:03:39.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:03:39.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:03:39.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:03:39.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:03:39.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:03:39.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:03:39.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:03:39.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:03:39.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:03:39.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:03:39.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:03:39.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:03:39.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:03:39.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:03:39.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:03:39.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:03:39.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:03:39.244 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.244 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.245 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.245 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:03:39.245 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:39.245 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 261.155039719, 2024-08-30T16:03:39.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:03:39.245 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.245 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:39.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:39.245 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:39.245 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:03:39.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:03:39.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:39.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:03:39.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:03:39.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:03:39.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:03:39.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:03:39.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:03:39.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:03:39.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:03:39.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:39.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:03:39.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:03:39.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:39.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:03:39.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:03:39.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:39.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:03:39.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:03:39.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:03:39.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:03:39.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.250 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.250 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.250 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.250 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:39.252 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:39.252 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:39.250+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:39.352 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:39.350+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:39.418 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:39.418 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:39.418 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:39.418 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:39.427 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:39.472 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:39.470+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:39.555 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:39.555 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:39.555 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:39.555 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:39.555 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:39.555 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:39.553+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:39.566 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:39.655 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:39.654+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:39.745 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.745 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 251.36344527899999, 2024-08-30T16:03:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:03:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:03:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:03:39.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:03:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:03:39.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:03:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:03:39.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:03:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:03:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:03:39.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:03:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:03:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:03:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:03:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:03:39.753 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 131.25044916300001, 2024-08-30T16:03:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:03:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:03:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:03:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:03:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:03:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:03:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:03:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:03:39.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:03:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:03:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:03:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:03:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:03:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.757 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-08-30T16:03:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:03:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 11.242029792, 2024-08-30T16:03:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037103309999999999, 2024-08-30T16:03:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 10 2024-08-30T16:03:39.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:03:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:03:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496549+0000", 2024-08-30T16:03:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:03:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496682+0000", 2024-08-30T16:03:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013371699999999999 2024-08-30T16:03:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496889+0000", 2024-08-30T16:03:39.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00020673600000000001 2024-08-30T16:03:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.497052+0000", 2024-08-30T16:03:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000162578 2024-08-30T16:03:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.498833+0000", 2024-08-30T16:03:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017810530000000001 2024-08-30T16:03:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:39.761 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.500282+0000", 2024-08-30T16:03:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001449437 2024-08-30T16:03:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:39.762 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:39.762 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:39.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:39.764 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:39.762+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:39.877 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:39.875+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:39.891 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:39.891 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:39.891 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:39.891 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:39.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:39.986 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:39.984+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:40.066 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:40.064+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:40.069 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:40.069 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:40.069 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:40.069 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:40.070 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:40.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:40.167 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:40.165+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:40.262 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:40.262 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:40.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:40.262 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:03:40.262 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:40.268 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:40.265+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:40.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:40.369 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:40.367+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:40.465 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:40.465 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:40.465 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:40.465 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:40.473 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:40.470+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:40.475 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:40.572 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:40.569+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:40.643 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:40.643 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:40.643 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:40.643 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:40.644 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:40.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:40.695 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:40.692+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:40.773 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:40.773+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:40.825 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:40.825 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:40.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:40.825 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:40.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.826 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:40.826 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:40.826 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 259.5506919, 2024-08-30T16:03:40.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:03:40.826 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:40.826 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:40.826 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:40.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:40.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:40.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:40.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:03:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:03:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:03:40.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:03:40.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:40.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:03:40.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:03:40.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:40.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:03:40.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:03:40.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:40.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:03:40.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:03:40.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:40.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:03:40.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:03:40.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:40.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:03:40.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:03:40.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:40.830 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:40.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:40.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.830 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:40.830 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:40.830 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 259.27776453899997, 2024-08-30T16:03:40.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:03:40.831 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:40.831 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:40.831 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:40.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:40.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:40.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:40.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:40.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:40.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:40.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:40.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:03:40.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:03:40.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:40.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:03:40.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:03:40.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:40.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:03:40.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:03:40.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:40.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:03:40.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:03:40.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:40.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:03:40.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:03:40.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:40.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:03:40.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:03:40.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:40.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:40.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:40.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:03:40.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:03:40.834 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:40.834 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:40.835 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:40.835 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:40.835 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:40.835 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:40.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:40.880 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:40.878+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:40.965 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:40.965 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:40.965 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:40.965 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:40.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:41.016 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:41.014+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:41.076 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:41.075+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:41.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:03:41.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-08-30T16:03:41.142 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:41.142 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:41.143 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:41.143 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:41.143 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:41.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:41.177 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:41.176+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:41.276 INFO:tasks.ceph.osd.6:Stopped 2024-08-30T16:03:41.276 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.6 2024-08-30T16:03:41.276 INFO:tasks.thrashosds.thrasher:remote for osd 6 is ubuntu@smithi094.front.sepia.ceph.com 2024-08-30T16:03:41.277 DEBUG:teuthology.orchestra.run.smithi094:> type ceph-objectstore-tool 2024-08-30T16:03:41.278 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:41.276+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:41.291 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-08-30T16:03:41.292 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2024-08-30T16:03:41.349 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:41.349 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:41.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:41.350 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:41.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:41.350 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:03:41.350 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:41.350 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 255.97228491600001, 2024-08-30T16:03:41.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:03:41.350 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:41.350 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:41.350 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:41.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:41.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:41.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:41.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:41.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:41.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:41.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:41.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:41.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:41.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:41.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:41.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:41.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:03:41.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:03:41.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:41.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:41.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:41.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:03:41.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:03:41.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:41.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:41.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:41.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:03:41.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:03:41.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:41.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:41.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:41.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:03:41.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:03:41.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:41.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:41.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:41.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:03:41.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:03:41.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:41.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:41.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:41.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:03:41.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:03:41.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:41.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:41.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:41.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:03:41.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:03:41.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:41.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:41.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:41.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:03:41.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:03:41.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:41.355 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:41.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:41.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:41.355 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:41.355 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:41.356 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:41.356 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:41.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:41.448 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:41.449 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:41.449 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:41.446+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:41.480 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:41.479+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:41.529 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:41.530 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:41.530 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:41.530 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:41.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:41.539 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:41.540 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:41.580 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:41.579+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:41.662 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:41.663 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:41.688 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:41.688+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:41.717 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:41.717 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:41.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:41.762 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:41.763 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:41.782 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:41.782+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:41.847 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:41.847 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:41.856 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:03:41.863 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:41.863 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:41.872 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:41.913 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:41.911+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:41.944 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:41.944 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:41.947 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:03:41.948 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:03:41.983 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:41.983+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:42.002 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:42.002 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:42.012 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:42.085 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:42.083+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:42.179 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:42.179 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:42.185 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:42.185+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:42.188 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:42.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:03:42.286 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:42.286+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:42.356 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:42.356 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:42.365 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:42.387 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:42.385+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:42.489 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:42.488+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:42.534 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:42.535 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:42.545 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:42.629 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:42.628+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:42.690 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:42.690+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:42.712 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:42.712 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:42.722 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:42.807 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:42.806+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:42.892 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:42.892 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:42.892 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:42.890+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:42.901 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:42.919 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-08-30T16:03:42.953 INFO:tasks.thrashosds.thrasher:No PGs found for osd.6 2024-08-30T16:03:42.989 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":46,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":38,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:03:42.993 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:42.991+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:43.068 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:43.068 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:43.078 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:43.122 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:43.121+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:43.195 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:43.195+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:43.205 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:43.206 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:43.215 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:43.299 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:43.298+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:43.379 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:43.380 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:43.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:43.430 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:43.429+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:43.499 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:43.499+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:43.536 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:43.536 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:43.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:43.599 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:43.599+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:43.681 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:43.681 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:43.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:43.700 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:43.699+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:43.801 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:43.800+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:43.835 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:43.835 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:43.847 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:43.902 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:43.899+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:43.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:03:43.993 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:43.993 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:44.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:44.003 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:44.000+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:44.104 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:44.102+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:44.175 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:44.175 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:44.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:44.195 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:44.195 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:44.239 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:44.236+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:44.287 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:44.287 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:44.306 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:44.303+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:44.328 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 263.054190861, 2024-08-30T16:03:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:03:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:03:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:03:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:03:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:03:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:03:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:03:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:03:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:03:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:03:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:03:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:03:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:03:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:03:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:03:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 262.78126350000002, 2024-08-30T16:03:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:03:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:03:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:03:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:03:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:03:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:03:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:03:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:03:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:03:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:03:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:03:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:03:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:03:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:03:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:03:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:44.339 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:44.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:44.376 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:44.377 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:44.407 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:44.406+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:44.474 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:44.475 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:44.507 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:44.507 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:44.507 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:44.507 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:44.507 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:44.505+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:44.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:44.560 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:44.561 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:44.609 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:44.606+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:44.651 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:44.652 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:03:44.685 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:44.685 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:44.685 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:44.685 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:44.685 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:44.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:44.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-08-30T16:03:44.710 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:44.709+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:44.737 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:44.737 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:44.740 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:03:44.741 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:03:44.811 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:44.808+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:44.866 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:44.866 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:44.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:44.866 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:44.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.866 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:03:44.867 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:44.867 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 259.48877202, 2024-08-30T16:03:44.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:03:44.867 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:44.867 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:44.867 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:44.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:44.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:44.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:44.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:44.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:44.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:44.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:44.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:03:44.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:03:44.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:44.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:03:44.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:03:44.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:44.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:03:44.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:03:44.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:44.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:03:44.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:03:44.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:44.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:03:44.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:03:44.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:44.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:03:44.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:03:44.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:44.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:03:44.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:03:44.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:44.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:44.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:44.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:03:44.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:03:44.871 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:44.871 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:44.871 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:44.871 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:44.871 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:44.871 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:44.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:44.920 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:44.916+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:44.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-08-30T16:03:45.012 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:45.010+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:45.068 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:45.068 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:45.068 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:45.068 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:45.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:45.118 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:45.116+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:45.214 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:45.214+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:45.244 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:45.244 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:45.244 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:45.244 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:45.244 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:45.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:45.316 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:45.313+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:45.416 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:45.414+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:45.429 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:45.429 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:45.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:45.429 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:45.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 260.95737336299999, 2024-08-30T16:03:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:03:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:03:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:03:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:03:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:03:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:03:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:03:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:03:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:03:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:03:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:03:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:03:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:03:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:03:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:03:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:03:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:03:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:03:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:03:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:03:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:03:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:03:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:03:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:03:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:03:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:03:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:03:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:03:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:03:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:03:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 259.84150702800002, 2024-08-30T16:03:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:03:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:45.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:45.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:45.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:03:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:03:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:03:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:03:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:03:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:03:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:03:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:03:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:03:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:03:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:03:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:03:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:03:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 259.78191552999999, 2024-08-30T16:03:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:03:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:03:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:03:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:03:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:03:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:03:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:03:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:03:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:03:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:03:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:03:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:03:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:03:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:03:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:03:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 259.75521684300003, 2024-08-30T16:03:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:03:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:03:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:03:45.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:03:45.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:03:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:03:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:03:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:03:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:03:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:03:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:03:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:03:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 258.88760031499999, 2024-08-30T16:03:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:03:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:03:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:45.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:03:45.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:03:45.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:45.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:03:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:03:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:03:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:03:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:03:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:03:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:03:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:03:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:03:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:03:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:03:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:03:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:03:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:03:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:03:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:03:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:03:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:03:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:03:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 258.72835323599998, 2024-08-30T16:03:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:03:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:03:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:03:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:03:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:03:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:03:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:03:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:03:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:03:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:03:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:03:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:03:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:03:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:03:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:03:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 258.45217917000002, 2024-08-30T16:03:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:03:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:03:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:03:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:03:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:03:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:03:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:03:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:03:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:03:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:03:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:03:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:03:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 257.79388759099999, 2024-08-30T16:03:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:03:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:03:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:03:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:03:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:03:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:03:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:03:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:03:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:03:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:03:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:03:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:03:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:03:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:03:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:03:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:03:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:03:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:03:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:03:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:03:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:03:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:03:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:03:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:03:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 257.69797430699998, 2024-08-30T16:03:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:03:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:03:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:03:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:03:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:03:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:03:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:03:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:03:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:03:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:03:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:03:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:03:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:03:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:03:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 198.44661379600001, 2024-08-30T16:03:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:03:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:03:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:03:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:03:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:03:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:03:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:03:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:03:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:03:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:03:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:03:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:03:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:03:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:03:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 138.43750910899999, 2024-08-30T16:03:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:03:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:03:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:03:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:03:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:03:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:03:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:03:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:45.487 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:45.488 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:45.530 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:45.527+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:45.618 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:45.618 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:45.618 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:45.618 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:45.618 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:45.617+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:45.627 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:45.720 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:45.719+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:45.750 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:45.751 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:45.751 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:45.751 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:45.751 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:45.761 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:45.777 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:07.313 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:45.820 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:45.818+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:45.901 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:45.901 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:45.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:45.901 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:03:45.901 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:45.910 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:45.921 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:45.919+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:46.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:03:46.004 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:07.541 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:46.022 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:46.020+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:46.035 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:46.035 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:46.035 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:46.035 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:46.045 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:46.123 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:46.120+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:46.126 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-08-30T16:03:46.216 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:46.216 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:46.216 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:46.216 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:46.216 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:46.223 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:46.221+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:46.228 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:46.324 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:46.322+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:46.425 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:46.424+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:46.442 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 275.578470004, 2024-08-30T16:03:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:03:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.443 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:03:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:03:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:03:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:03:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:03:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:03:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:03:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:03:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:03:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:03:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:03:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:03:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:03:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:03:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:03:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:03:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:03:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:03:46.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:03:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:03:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:03:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:03:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:03:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:03:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 274.66205398400001, 2024-08-30T16:03:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:03:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.449 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:03:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:46.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:03:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:03:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:03:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:03:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:03:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:03:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:03:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:03:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:03:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:03:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:03:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:03:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:03:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:03:46.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:03:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:03:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:03:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:46.454 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 274.66034943099999, 2024-08-30T16:03:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:03:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:03:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:03:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:03:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:03:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:03:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:03:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:03:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:03:46.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:03:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:03:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:03:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:03:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:03:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:03:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:03:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:03:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:03:46.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:03:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:03:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 274.24089562400002, 2024-08-30T16:03:46.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:03:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:03:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:03:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:03:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:03:46.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:03:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:03:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:03:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:03:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:03:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:03:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:03:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:03:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:03:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:03:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:03:46.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:03:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:03:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:03:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:03:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 274.22400744999999, 2024-08-30T16:03:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:03:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:03:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:03:46.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:03:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:03:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:03:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:03:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:03:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:03:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:03:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:03:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:03:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:03:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:03:46.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:03:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:03:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:03:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:03:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:03:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:03:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 274.211007381, 2024-08-30T16:03:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:03:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.471 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:03:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:03:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:03:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:03:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:03:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:03:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:03:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:03:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:03:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:03:46.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:03:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:03:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:03:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:03:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:03:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:03:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:03:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:03:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:03:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.476 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 274.208515089, 2024-08-30T16:03:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:03:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:03:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:46.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:03:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:03:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:03:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:03:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:03:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:03:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:03:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:03:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:03:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:03:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:03:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:03:46.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:03:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:03:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:03:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:03:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:03:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 274.207757641, 2024-08-30T16:03:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:03:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:03:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.482 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:03:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:03:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:03:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:03:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:03:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:03:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:03:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:03:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:03:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:03:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:03:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:03:46.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:03:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:03:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:03:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:03:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:03:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:03:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 274.20702519100001, 2024-08-30T16:03:46.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:03:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:03:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:46.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:03:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:03:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:03:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:03:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:03:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:03:46.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:03:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:03:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:03:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:03:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:03:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:03:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:03:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:03:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:03:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:03:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:03:46.492 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 274.20448639199998, 2024-08-30T16:03:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:03:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.493 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:03:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:03:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:03:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:03:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:03:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:03:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:03:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:03:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:03:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:03:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:03:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:03:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:03:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:03:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:03:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:03:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:03:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:03:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 274.203738391, 2024-08-30T16:03:46.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:03:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:03:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:03:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:03:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:03:46.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:03:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:03:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:03:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:03:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:03:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:03:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:03:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:03:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:03:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:03:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:03:46.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:03:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:03:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:03:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:03:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 273.43162191900001, 2024-08-30T16:03:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:03:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:03:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:03:46.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:03:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:03:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:03:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:03:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:03:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:03:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:03:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:03:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:03:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:03:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:03:46.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:03:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:03:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:03:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:03:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:03:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:03:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 273.42956477299998, 2024-08-30T16:03:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:03:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:03:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.510 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:03:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:03:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:03:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:03:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:03:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:03:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:03:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:03:46.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:03:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:03:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:03:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:03:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:03:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:03:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:03:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:03:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:03:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.514 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 273.16662027299998, 2024-08-30T16:03:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:03:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.515 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:03:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:03:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:03:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:03:46.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:03:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:03:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:03:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:03:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:03:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:03:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:03:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:03:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:03:46.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:03:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:03:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:03:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:03:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:03:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:03:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 273.16552876999998, 2024-08-30T16:03:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:03:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.523 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:03:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:03:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:03:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:03:46.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:03:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:03:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:03:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:03:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:03:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:03:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:03:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:03:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:03:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:03:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:03:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:03:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:03:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:03:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 273.16477918499999, 2024-08-30T16:03:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:03:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:03:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:03:46.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:03:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:03:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:03:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:03:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:03:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:03:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:03:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:03:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:03:46.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:03:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:03:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:03:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:03:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:03:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:03:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:03:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 273.131047653, 2024-08-30T16:03:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:03:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:03:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:03:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:03:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:03:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:03:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:03:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:03:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:03:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:03:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:03:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:03:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:03:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:03:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:03:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:03:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:03:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:03:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:03:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:03:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 273.13052164499999, 2024-08-30T16:03:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:03:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:03:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:03:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:03:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:03:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:03:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:03:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:03:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:03:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:03:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:03:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:03:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:03:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:03:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:03:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:03:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:03:46.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:03:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:03:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 273.13006608199998, 2024-08-30T16:03:46.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:03:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:03:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:03:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:03:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:03:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:03:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:03:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:03:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:03:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:03:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:03:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:03:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:03:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:03:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:03:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:03:46.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:03:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:03:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:03:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:03:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:03:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 268.45955076600001, 2024-08-30T16:03:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:03:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:03:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.551 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:03:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:03:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:03:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:03:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:03:46.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:03:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:03:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:03:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:03:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:46.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:03:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:03:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:03:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:03:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:03:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:03:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:46.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:03:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:03:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:46.556 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:46.556 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:46.558 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:46.558 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:46.556+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:46.658 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:46.656+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:46.723 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:46.723 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:46.723 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:46.723 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:46.733 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:46.736 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":46,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":41,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:03:46.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:03:46.790 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:46.788+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:46.858 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:46.858 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:46.858 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:46.860 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:46.857+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:46.868 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:46.961 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:46.959+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:46.977 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:46.998 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:46.999 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:46.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:46.999 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:46.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:46.999 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:46.999 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:46.999 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 258.61715240500001, 2024-08-30T16:03:46.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:03:46.999 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:46.999 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:46.999 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:46.999 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:47.000 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:47.000 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:03:47.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.000 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:47.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:47.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:47.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:47.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.000 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:47.000 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:03:47.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:47.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:47.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:03:47.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:03:47.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.001 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:47.001 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:03:47.001 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:03:47.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:47.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:03:47.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:03:47.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:47.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:03:47.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:03:47.002 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.002 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.002 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:47.002 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:03:47.002 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:03:47.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.003 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:47.003 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:03:47.003 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:03:47.003 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:47.003 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:47.003 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:47.003 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.003 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.003 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:03:47.003 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:47.003 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 138.50415628900001, 2024-08-30T16:03:47.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:03:47.004 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:47.004 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:47.004 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:47.004 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:47.004 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:47.004 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:03:47.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.004 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:47.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:47.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:47.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:47.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:47.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:03:47.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:47.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:47.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:03:47.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:47.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:47.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:03:47.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:03:47.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:47.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:03:47.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:03:47.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:47.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:03:47.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:03:47.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:47.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:03:47.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:03:47.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:47.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:03:47.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:03:47.007 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:47.007 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:47.007 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:47.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.008 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-08-30T16:03:47.008 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:03:47.008 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.495736917999999, 2024-08-30T16:03:47.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037103309999999999, 2024-08-30T16:03:47.008 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:47.008 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:03:47.008 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:47.008 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:03:47.008 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:47.008 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 10 2024-08-30T16:03:47.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.008 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:47.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:47.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:03:47.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:47.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:47.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:03:47.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:47.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.009 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:47.009 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496549+0000", 2024-08-30T16:03:47.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:03:47.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:47.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496682+0000", 2024-08-30T16:03:47.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013371699999999999 2024-08-30T16:03:47.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:47.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496889+0000", 2024-08-30T16:03:47.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00020673600000000001 2024-08-30T16:03:47.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:47.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.497052+0000", 2024-08-30T16:03:47.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000162578 2024-08-30T16:03:47.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:47.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.498833+0000", 2024-08-30T16:03:47.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017810530000000001 2024-08-30T16:03:47.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:47.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:47.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:47.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.500282+0000", 2024-08-30T16:03:47.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001449437 2024-08-30T16:03:47.011 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:47.012 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:47.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:47.012 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:47.012 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:47.012 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:47.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:47.062 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:47.060+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:47.096 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:47.097 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:47.151 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:47.151 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:47.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:47.192 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:47.192 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:47.196 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:47.193+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:47.264 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:47.262+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:47.319 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:47.319 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:47.337 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:47.337 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:47.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:47.388 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:47.387+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:47.403 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:47.403 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:47.466 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:47.463+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:47.485 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:47.485 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:47.486 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:03:47.517 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:47.518 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:47.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:47.573 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:47.570+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:47.578 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:47.578 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:47.581 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:03:47.582 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:03:47.667 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:47.666+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:47.701 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:47.701 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:47.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:47.769 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:47.767+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:47.849 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:47.849 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:47.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:47.870 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:47.868+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:47.955 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7] out_osds: [6, 4, 2] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-08-30T16:03:47.955 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:03:47.955 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-30T16:03:47.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-08-30T16:03:47.970 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:47.969+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:48.049 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:48.049 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:48.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:48.071 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:48.070+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:48.141 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-08-30T16:03:48.187 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:48.184+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:48.199 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:48.199 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:48.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:48.274 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:48.271+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:48.374 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:48.372+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:48.398 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:48.398 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:48.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:48.493 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:48.491+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:48.578 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:48.576+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:48.603 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:48.604 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:48.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:48.676 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:03:48.676 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:48.676 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 80, 2024-08-30T16:03:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "9d72e2e2-0554-42af-8e62-9b1a3a804112", 2024-08-30T16:03:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-08-30T15:58:20.129410+0000", 2024-08-30T16:03:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-08-30T16:03:47.921801+0000", 2024-08-30T16:03:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "last_up_change": "2024-08-30T16:03:35.238525+0000", 2024-08-30T16:03:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "last_in_change": "2024-08-30T16:03:05.847464+0000", 2024-08-30T16:03:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T16:03:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_num": 5804032, 2024-08-30T16:03:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_set": [ 2024-08-30T16:03:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "nodeep-scrub", 2024-08-30T16:03:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "pglog_hardlimit", 2024-08-30T16:03:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "purged_snapdirs", 2024-08-30T16:03:48.677 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_deletes", 2024-08-30T16:03:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "sortbitwise" 2024-08-30T16:03:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:03:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_version": 7, 2024-08-30T16:03:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T16:03:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T16:03:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T16:03:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_snapshot": "", 2024-08-30T16:03:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_max": 2, 2024-08-30T16:03:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd": 8, 2024-08-30T16:03:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "require_min_compat_client": "luminous", 2024-08-30T16:03:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "min_compat_client": "luminous", 2024-08-30T16:03:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "require_osd_release": "reef", 2024-08-30T16:03:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_crimson": false, 2024-08-30T16:03:48.678 INFO:teuthology.orchestra.run.smithi050.stdout: "pools": [ 2024-08-30T16:03:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 1, 2024-08-30T16:03:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".mgr", 2024-08-30T16:03:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-08-30T15:59:08.355540+0000", 2024-08-30T16:03:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-08-30T16:03:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-08-30T16:03:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-08-30T16:03:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 2, 2024-08-30T16:03:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 1, 2024-08-30T16:03:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-08-30T16:03:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-08-30T16:03:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-08-30T16:03:48.679 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T16:03:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T16:03:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-08-30T16:03:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-08-30T16:03:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 1, 2024-08-30T16:03:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 1, 2024-08-30T16:03:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 1, 2024-08-30T16:03:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 1, 2024-08-30T16:03:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 1, 2024-08-30T16:03:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-08-30T16:03:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-08-30T16:03:48.680 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-08-30T16:03:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-08-30T16:03:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-08-30T16:03:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-08-30T16:03:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-08-30T16:03:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "19", 2024-08-30T16:03:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-08-30T16:03:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T16:03:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T16:03:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-08-30T16:03:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-08-30T16:03:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-08-30T16:03:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-08-30T16:03:48.681 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-08-30T16:03:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-08-30T16:03:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-08-30T16:03:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-08-30T16:03:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-08-30T16:03:48.682 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-08-30T16:03:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-08-30T16:03:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-08-30T16:03:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-08-30T16:03:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-08-30T16:03:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-08-30T16:03:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T16:03:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T16:03:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T16:03:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-08-30T16:03:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-08-30T16:03:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-08-30T16:03:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-08-30T16:03:48.683 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-08-30T16:03:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-08-30T16:03:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-08-30T16:03:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-08-30T16:03:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-08-30T16:03:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-08-30T16:03:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T16:03:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-08-30T16:03:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-08-30T16:03:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-08-30T16:03:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-08-30T16:03:48.684 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-08-30T16:03:48.685 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-08-30T16:03:48.685 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_max": 32, 2024-08-30T16:03:48.685 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_min": 1 2024-08-30T16:03:48.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.685 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-08-30T16:03:48.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": {} 2024-08-30T16:03:48.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.685 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-08-30T16:03:48.685 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 8, 2024-08-30T16:03:48.685 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 8, 2024-08-30T16:03:48.685 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.25, 2024-08-30T16:03:48.685 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-08-30T16:03:48.685 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-08-30T16:03:48.685 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-08-30T16:03:48.686 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-08-30T16:03:48.686 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-08-30T16:03:48.686 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.686 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-08-30T16:03:48.686 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "rbd", 2024-08-30T16:03:48.686 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-08-30T15:59:21.696796+0000", 2024-08-30T16:03:48.686 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-08-30T16:03:48.686 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-08-30T16:03:48.686 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-08-30T16:03:48.686 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 2, 2024-08-30T16:03:48.686 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 1, 2024-08-30T16:03:48.687 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-08-30T16:03:48.687 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-08-30T16:03:48.687 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-08-30T16:03:48.687 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T16:03:48.687 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T16:03:48.687 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-08-30T16:03:48.687 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-08-30T16:03:48.687 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 8, 2024-08-30T16:03:48.687 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 8, 2024-08-30T16:03:48.687 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 8, 2024-08-30T16:03:48.687 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 8, 2024-08-30T16:03:48.687 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 8, 2024-08-30T16:03:48.687 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-08-30T16:03:48.687 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-08-30T16:03:48.688 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-08-30T16:03:48.688 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-08-30T16:03:48.688 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-08-30T16:03:48.688 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-08-30T16:03:48.688 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-08-30T16:03:48.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.688 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "28", 2024-08-30T16:03:48.688 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-08-30T16:03:48.688 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T16:03:48.688 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T16:03:48.688 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-08-30T16:03:48.688 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-08-30T16:03:48.688 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 3, 2024-08-30T16:03:48.689 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 28, 2024-08-30T16:03:48.689 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-08-30T16:03:48.689 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-08-30T16:03:48.689 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-08-30T16:03:48.689 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-08-30T16:03:48.689 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-08-30T16:03:48.689 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-08-30T16:03:48.689 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-08-30T16:03:48.689 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-08-30T16:03:48.689 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-08-30T16:03:48.689 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-08-30T16:03:48.689 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-08-30T16:03:48.689 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T16:03:48.690 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T16:03:48.690 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T16:03:48.690 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-08-30T16:03:48.690 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-08-30T16:03:48.690 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-08-30T16:03:48.690 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-08-30T16:03:48.690 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-08-30T16:03:48.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.690 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-08-30T16:03:48.690 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-08-30T16:03:48.690 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-08-30T16:03:48.690 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-08-30T16:03:48.690 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-08-30T16:03:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T16:03:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-08-30T16:03:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-08-30T16:03:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-08-30T16:03:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-08-30T16:03:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-08-30T16:03:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-08-30T16:03:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-08-30T16:03:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd": {} 2024-08-30T16:03:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-08-30T16:03:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 4.9699997901916504, 2024-08-30T16:03:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 4.9699997901916504, 2024-08-30T16:03:48.691 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.62999999523162842, 2024-08-30T16:03:48.692 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 3.130000114440918, 2024-08-30T16:03:48.692 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 3.130000114440918, 2024-08-30T16:03:48.692 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-08-30T16:03:48.692 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-08-30T16:03:48.692 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-08-30T16:03:48.692 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.692 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.692 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:03:48.693 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-08-30T16:03:48.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.693 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-08-30T16:03:48.693 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4", 2024-08-30T16:03:48.693 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:48.693 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:48.693 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:48.693 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:48.693 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:48.693 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:48.694 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 15, 2024-08-30T16:03:48.694 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 60, 2024-08-30T16:03:48.694 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:48.694 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:48.694 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:48.694 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.694 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.694 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6812", 2024-08-30T16:03:48.694 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:03:48.694 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.694 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.694 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:48.695 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.695 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.695 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6813", 2024-08-30T16:03:48.695 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:03:48.695 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.695 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.695 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6815", 2024-08-30T16:03:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:03:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6814", 2024-08-30T16:03:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:03:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6812/1957399494", 2024-08-30T16:03:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6813/1957399494", 2024-08-30T16:03:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6815/1957399494", 2024-08-30T16:03:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6814/1957399494", 2024-08-30T16:03:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-08-30T16:03:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "0fe11a1c-4326-4bfd-a2a4-bff14023f301", 2024-08-30T16:03:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 22, 2024-08-30T16:03:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 22, 2024-08-30T16:03:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6808", 2024-08-30T16:03:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:03:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6809", 2024-08-30T16:03:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:03:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6811", 2024-08-30T16:03:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:03:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6810", 2024-08-30T16:03:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:03:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6808/3208178756", 2024-08-30T16:03:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6809/3208178756", 2024-08-30T16:03:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6811/3208178756", 2024-08-30T16:03:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6810/3208178756", 2024-08-30T16:03:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-08-30T16:03:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "a9d5354b-3600-41ed-80c5-a3439da7aca6", 2024-08-30T16:03:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:03:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:03:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 16, 2024-08-30T16:03:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:03:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6804", 2024-08-30T16:03:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:03:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6805", 2024-08-30T16:03:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:03:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6807", 2024-08-30T16:03:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:03:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6806", 2024-08-30T16:03:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:03:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6804/287018800", 2024-08-30T16:03:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6805/287018800", 2024-08-30T16:03:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6807/287018800", 2024-08-30T16:03:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6806/287018800", 2024-08-30T16:03:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-08-30T16:03:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "78bcc398-a7e4-4efc-9836-1dfb94407c9c", 2024-08-30T16:03:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:03:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 64, 2024-08-30T16:03:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6800", 2024-08-30T16:03:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:03:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6801", 2024-08-30T16:03:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:03:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6803", 2024-08-30T16:03:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:03:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6802", 2024-08-30T16:03:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:03:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6800/1140717714", 2024-08-30T16:03:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6801/1140717714", 2024-08-30T16:03:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6803/1140717714", 2024-08-30T16:03:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6802/1140717714", 2024-08-30T16:03:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-08-30T16:03:48.712 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "53a1060b-7985-4c2c-820e-64b158056113", 2024-08-30T16:03:48.712 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:48.712 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:03:48.712 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:03:48.712 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:48.712 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:48.712 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:48.712 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 20, 2024-08-30T16:03:48.712 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 24, 2024-08-30T16:03:48.712 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6805", 2024-08-30T16:03:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:03:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6806", 2024-08-30T16:03:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:03:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6808", 2024-08-30T16:03:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:03:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6807", 2024-08-30T16:03:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:03:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6805/1468073345", 2024-08-30T16:03:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6806/1468073345", 2024-08-30T16:03:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6808/1468073345", 2024-08-30T16:03:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6807/1468073345", 2024-08-30T16:03:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-08-30T16:03:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "853a03f7-227d-4f98-9e5c-c29896820746", 2024-08-30T16:03:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 13, 2024-08-30T16:03:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 64, 2024-08-30T16:03:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6809", 2024-08-30T16:03:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:03:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6810", 2024-08-30T16:03:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:03:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6812", 2024-08-30T16:03:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:03:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6811", 2024-08-30T16:03:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:03:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6809/3876025616", 2024-08-30T16:03:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6810/3876025616", 2024-08-30T16:03:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6812/3876025616", 2024-08-30T16:03:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6811/3876025616", 2024-08-30T16:03:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-08-30T16:03:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "bc978060-bf28-4027-8bba-7d6d8659bfc5", 2024-08-30T16:03:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 0, 2024-08-30T16:03:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:03:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:03:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:03:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:03:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 75, 2024-08-30T16:03:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6801", 2024-08-30T16:03:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:03:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.723 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:48.723 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.723 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.723 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6802", 2024-08-30T16:03:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:03:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6804", 2024-08-30T16:03:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:03:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6803", 2024-08-30T16:03:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:03:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6801/128519122", 2024-08-30T16:03:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6802/128519122", 2024-08-30T16:03:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6804/128519122", 2024-08-30T16:03:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6803/128519122", 2024-08-30T16:03:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "exists" 2024-08-30T16:03:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-08-30T16:03:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "45df5c6f-b124-46d1-9efb-62b78bc584ed", 2024-08-30T16:03:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:03:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:03:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:03:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:03:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:03:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:03:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:03:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 40, 2024-08-30T16:03:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:03:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:03:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:03:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6813", 2024-08-30T16:03:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:03:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:03:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6814", 2024-08-30T16:03:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:03:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:03:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6816", 2024-08-30T16:03:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:03:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:03:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:03:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:03:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6815", 2024-08-30T16:03:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:03:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6813/68721277", 2024-08-30T16:03:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6814/68721277", 2024-08-30T16:03:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6816/68721277", 2024-08-30T16:03:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6815/68721277", 2024-08-30T16:03:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:03:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:03:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:03:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:03:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_xinfo": [ 2024-08-30T16:03:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-08-30T16:03:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.911895+0000", 2024-08-30T16:03:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-08-30T16:03:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:04.786543+0000", 2024-08-30T16:03:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-08-30T16:03:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:03:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:04.197952+0000", 2024-08-30T16:03:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-08-30T16:03:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.680847+0000", 2024-08-30T16:03:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-08-30T16:03:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:03:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:01.937600+0000", 2024-08-30T16:03:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-08-30T16:03:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:02.082761+0000", 2024-08-30T16:03:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-08-30T16:03:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:03:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.401025+0000", 2024-08-30T16:03:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 74 2024-08-30T16:03:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-08-30T16:03:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:03:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:03:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:03:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:03:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:03:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.129371+0000", 2024-08-30T16:03:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:03:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:03:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap": [ 2024-08-30T16:03:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "2.7", 2024-08-30T16:03:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-08-30T16:03:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: 4, 2024-08-30T16:03:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: 1 2024-08-30T16:03:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:03:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_items": [], 2024-08-30T16:03:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_primaries": [], 2024-08-30T16:03:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_temp": [], 2024-08-30T16:03:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_temp": [], 2024-08-30T16:03:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklist": {}, 2024-08-30T16:03:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "range_blocklist": {}, 2024-08-30T16:03:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profiles": { 2024-08-30T16:03:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "default": { 2024-08-30T16:03:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-08-30T16:03:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-08-30T16:03:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-08-30T16:03:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-08-30T16:03:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van" 2024-08-30T16:03:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps_queue": [], 2024-08-30T16:03:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "new_removed_snaps": [], 2024-08-30T16:03:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "new_purged_snaps": [], 2024-08-30T16:03:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_node_flags": {}, 2024-08-30T16:03:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "device_class_flags": {}, 2024-08-30T16:03:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": { 2024-08-30T16:03:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_enabled": false, 2024-08-30T16:03:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_bucket_count": 0, 2024-08-30T16:03:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "degraded_stretch_mode": 0, 2024-08-30T16:03:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "recovering_stretch_mode": 0, 2024-08-30T16:03:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_bucket": 0 2024-08-30T16:03:48.740 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:48.740 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:48.741 INFO:tasks.thrashosds.thrasher:j is {'epoch': 80, 'fsid': '9d72e2e2-0554-42af-8e62-9b1a3a804112', 'created': '2024-08-30T15:58:20.129410+0000', 'modified': '2024-08-30T16:03:47.921801+0000', 'last_up_change': '2024-08-30T16:03:35.238525+0000', 'last_in_change': '2024-08-30T16:03:05.847464+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T15:59:08.355540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-08-30T15:59:21.696796+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.96999979019165, 'score_stable': 4.96999979019165, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 3.130000114440918, 'raw_score_stable': 3.130000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 60, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6812', 'nonce': 1957399494}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6813', 'nonce': 1957399494}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 1957399494}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6814', 'nonce': 1957399494}]}, 'public_addr': '172.21.15.50:6812/1957399494', 'cluster_addr': '172.21.15.50:6813/1957399494', 'heartbeat_back_addr': '172.21.15.50:6815/1957399494', 'heartbeat_front_addr': '172.21.15.50:6814/1957399494', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0fe11a1c-4326-4bfd-a2a4-bff14023f301', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6808', 'nonce': 3208178756}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6809', 'nonce': 3208178756}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 3208178756}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6810', 'nonce': 3208178756}]}, 'public_addr': '172.21.15.50:6808/3208178756', 'cluster_addr': '172.21.15.50:6809/3208178756', 'heartbeat_back_addr': '172.21.15.50:6811/3208178756', 'heartbeat_front_addr': '172.21.15.50:6810/3208178756', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a9d5354b-3600-41ed-80c5-a3439da7aca6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6804', 'nonce': 287018800}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6805', 'nonce': 287018800}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 287018800}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6806', 'nonce': 287018800}]}, 'public_addr': '172.21.15.50:6804/287018800', 'cluster_addr': '172.21.15.50:6805/287018800', 'heartbeat_back_addr': '172.21.15.50:6807/287018800', 'heartbeat_front_addr': '172.21.15.50:6806/287018800', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '78bcc398-a7e4-4efc-9836-1dfb94407c9c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 64, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6800', 'nonce': 1140717714}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6801', 'nonce': 1140717714}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 1140717714}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6802', 'nonce': 1140717714}]}, 'public_addr': '172.21.15.50:6800/1140717714', 'cluster_addr': '172.21.15.50:6801/1140717714', 'heartbeat_back_addr': '172.21.15.50:6803/1140717714', 'heartbeat_front_addr': '172.21.15.50:6802/1140717714', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '53a1060b-7985-4c2c-820e-64b158056113', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 1468073345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6806', 'nonce': 1468073345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6808', 'nonce': 1468073345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 1468073345}]}, 'public_addr': '172.21.15.94:6805/1468073345', 'cluster_addr': '172.21.15.94:6806/1468073345', 'heartbeat_back_addr': '172.21.15.94:6808/1468073345', 'heartbeat_front_addr': '172.21.15.94:6807/1468073345', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '853a03f7-227d-4f98-9e5c-c29896820746', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 64, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 3876025616}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6810', 'nonce': 3876025616}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6812', 'nonce': 3876025616}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 3876025616}]}, 'public_addr': '172.21.15.94:6809/3876025616', 'cluster_addr': '172.21.15.94:6810/3876025616', 'heartbeat_back_addr': '172.21.15.94:6812/3876025616', 'heartbeat_front_addr': '172.21.15.94:6811/3876025616', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bc978060-bf28-4027-8bba-7d6d8659bfc5', 'up': 0, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 0, 'down_at': 75, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 128519122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6802', 'nonce': 128519122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6804', 'nonce': 128519122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 128519122}]}, 'public_addr': '172.21.15.94:6801/128519122', 'cluster_addr': '172.21.15.94:6802/128519122', 'heartbeat_back_addr': '172.21.15.94:6804/128519122', 'heartbeat_front_addr': '172.21.15.94:6803/128519122', 'state': ['exists']}, {'osd': 7, 'uuid': '45df5c6f-b124-46d1-9efb-62b78bc584ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 68721277}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6814', 'nonce': 68721277}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6816', 'nonce': 68721277}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 68721277}]}, 'public_addr': '172.21.15.94:6813/68721277', 'cluster_addr': '172.21.15.94:6814/68721277', 'heartbeat_back_addr': '172.21.15.94:6816/68721277', 'heartbeat_front_addr': '172.21.15.94:6815/68721277', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.911895+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:04.786543+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:04.197952+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.680847+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:01.937600+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:02.082761+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.401025+0000', 'dead_epoch': 74}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.129371+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.7', 'osds': [4, 1]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T16:03:48.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:03:48.742 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:48.739+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:48.784 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:48.784 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:48.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:48.879 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:48.877+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:48.926 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:10.463 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:48.943 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:48.942+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:48.967 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:48.967 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:48.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:49.035 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:10.572 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:49.062 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:49.060+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:49.146 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:49.144+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:49.159 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:49.160 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:49.169 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:49.247 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:49.245+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:49.312 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:49.312 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:49.321 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:49.348 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:49.347+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:49.393 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:03:49.394 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:03:49.406 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-08-30T16:03:48.021957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":397859,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":137924,"kb_used_data":2824,"kb_used_omap":7,"kb_used_meta":135032,"kb_avail":471721276,"statfs":{"total":483183820800,"available":483042586624,"internally_reserved":0,"allocated":2891776,"data_stored":2206752,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":7967,"internal_metadata":138272993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.203378"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-08-30T16:03:43.680977+0000","last_change":"2024-08-30T15:59:27.646756+0000","last_active":"2024-08-30T16:03:43.680977+0000","last_peered":"2024-08-30T16:03:43.680977+0000","last_clean":"2024-08-30T16:03:43.680977+0000","last_became_active":"2024-08-30T15:59:25.342321+0000","last_became_peered":"2024-08-30T15:59:25.342321+0000","last_unstale":"2024-08-30T16:03:43.680977+0000","last_undegraded":"2024-08-30T16:03:43.680977+0000","last_fullsized":"2024-08-30T16:03:43.680977+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:53:42.927368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.116914037,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-08-30T16:03:40.255360+0000","last_change":"2024-08-30T16:02:40.613055+0000","last_active":"2024-08-30T16:03:40.255360+0000","last_peered":"2024-08-30T16:03:40.255360+0000","last_clean":"2024-08-30T16:03:40.255360+0000","last_became_active":"2024-08-30T16:02:40.576990+0000","last_became_peered":"2024-08-30T16:02:40.576990+0000","last_unstale":"2024-08-30T16:03:40.255360+0000","last_undegraded":"2024-08-30T16:03:40.255360+0000","last_fullsized":"2024-08-30T16:03:40.255360+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T01:33:27.937690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":75,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-08-30T16:03:40.255288+0000","last_change":"2024-08-30T16:02:40.454608+0000","last_active":"2024-08-30T16:03:40.255288+0000","last_peered":"2024-08-30T16:03:40.255288+0000","last_clean":"2024-08-30T16:03:40.255288+0000","last_became_active":"2024-08-30T16:02:40.448507+0000","last_became_peered":"2024-08-30T16:02:40.448507+0000","last_unstale":"2024-08-30T16:03:40.255288+0000","last_undegraded":"2024-08-30T16:03:40.255288+0000","last_fullsized":"2024-08-30T16:03:40.255288+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T00:30:34.065701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":81,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-08-30T16:03:39.701320+0000","last_change":"2024-08-30T16:03:08.888511+0000","last_active":"2024-08-30T16:03:39.701320+0000","last_peered":"2024-08-30T16:03:39.701320+0000","last_clean":"2024-08-30T16:03:39.701320+0000","last_became_active":"2024-08-30T16:03:08.882505+0000","last_became_peered":"2024-08-30T16:03:08.882505+0000","last_unstale":"2024-08-30T16:03:39.701320+0000","last_undegraded":"2024-08-30T16:03:39.701320+0000","last_fullsized":"2024-08-30T16:03:39.701320+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:06:24.414887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":81,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-08-30T16:03:40.250248+0000","last_change":"2024-08-30T16:01:01.435839+0000","last_active":"2024-08-30T16:03:40.250248+0000","last_peered":"2024-08-30T16:03:40.250248+0000","last_clean":"2024-08-30T16:03:40.250248+0000","last_became_active":"2024-08-30T16:01:00.974833+0000","last_became_peered":"2024-08-30T16:01:00.974833+0000","last_unstale":"2024-08-30T16:03:40.250248+0000","last_undegraded":"2024-08-30T16:03:40.250248+0000","last_fullsized":"2024-08-30T16:03:40.250248+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:09:38.575833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":65,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-08-30T16:03:43.486892+0000","last_change":"2024-08-30T15:59:27.825532+0000","last_active":"2024-08-30T16:03:43.486892+0000","last_peered":"2024-08-30T16:03:43.486892+0000","last_clean":"2024-08-30T16:03:43.486892+0000","last_became_active":"2024-08-30T15:59:25.469390+0000","last_became_peered":"2024-08-30T15:59:25.469390+0000","last_unstale":"2024-08-30T16:03:43.486892+0000","last_undegraded":"2024-08-30T16:03:43.486892+0000","last_fullsized":"2024-08-30T16:03:43.486892+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T17:46:39.688911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":67,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-08-30T16:03:40.247460+0000","last_change":"2024-08-30T15:59:27.583986+0000","last_active":"2024-08-30T16:03:40.247460+0000","last_peered":"2024-08-30T16:03:40.247460+0000","last_clean":"2024-08-30T16:03:40.247460+0000","last_became_active":"2024-08-30T15:59:24.528536+0000","last_became_peered":"2024-08-30T15:59:24.528536+0000","last_unstale":"2024-08-30T16:03:40.247460+0000","last_undegraded":"2024-08-30T16:03:40.247460+0000","last_fullsized":"2024-08-30T16:03:40.247460+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:18:40.602710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-08-30T16:03:40.253446+0000","last_change":"2024-08-30T16:03:08.552899+0000","last_active":"2024-08-30T16:03:40.253446+0000","last_peered":"2024-08-30T16:03:40.253446+0000","last_clean":"2024-08-30T16:03:40.253446+0000","last_became_active":"2024-08-30T16:03:08.491323+0000","last_became_peered":"2024-08-30T16:03:08.491323+0000","last_unstale":"2024-08-30T16:03:40.253446+0000","last_undegraded":"2024-08-30T16:03:40.253446+0000","last_fullsized":"2024-08-30T16:03:40.253446+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:26:44.292348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":68,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-08-30T16:03:40.250172+0000","last_change":"2024-08-30T15:59:21.623783+0000","last_active":"2024-08-30T16:03:40.250172+0000","last_peered":"2024-08-30T16:03:40.250172+0000","last_clean":"2024-08-30T16:03:40.250172+0000","last_became_active":"2024-08-30T15:59:20.848924+0000","last_became_peered":"2024-08-30T15:59:20.848924+0000","last_unstale":"2024-08-30T16:03:40.250172+0000","last_undegraded":"2024-08-30T16:03:40.250172+0000","last_fullsized":"2024-08-30T16:03:40.250172+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:14:52.575421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":795680,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280574,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":606208,"data_stored":683521,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":12.510999999999999,"5min":12.433,"15min":13.757999999999999},"min":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"max":{"1min":51.027000000000001,"5min":57.177999999999997,"15min":57.177999999999997},"last":10.335000000000001},{"interface":"front","average":{"1min":13.460000000000001,"5min":12.444000000000001,"15min":13.041},"min":{"1min":6.883,"5min":5.0510000000000002,"15min":5.0510000000000002},"max":{"1min":54.682000000000002,"5min":58.024999999999999,"15min":58.024999999999999},"last":6.327}]},{"osd":2,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5579999999999998,"5min":10.762,"15min":10.539999999999999},"min":{"1min":5.5780000000000003,"5min":5.5780000000000003,"15min":5.5780000000000003},"max":{"1min":20.256,"5min":73.399000000000001,"15min":73.399000000000001},"last":7.1970000000000001},{"interface":"front","average":{"1min":14.210000000000001,"5min":14.776999999999999,"15min":14.977},"min":{"1min":8.4920000000000009,"5min":6.9139999999999997,"15min":6.9139999999999997},"max":{"1min":26.715,"5min":75.120000000000005,"15min":75.120000000000005},"last":10.867000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":17.975999999999999,"5min":16.571999999999999,"15min":17.300000000000001},"min":{"1min":8.4849999999999994,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":66.869,"5min":66.869,"15min":66.869},"last":12.196999999999999},{"interface":"front","average":{"1min":13.228,"5min":14.532,"15min":14.795999999999999},"min":{"1min":7.9930000000000003,"5min":7.6619999999999999,"15min":7.6619999999999999},"max":{"1min":29.524999999999999,"5min":104.38200000000001,"15min":104.38200000000001},"last":14.694000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":17.800000000000001,"5min":15.215,"15min":14.923},"min":{"1min":6.7279999999999998,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":66.340999999999994,"5min":66.340999999999994,"15min":66.340999999999994},"last":15.738},{"interface":"front","average":{"1min":13.164999999999999,"5min":13.124000000000001,"15min":13.084},"min":{"1min":7.8970000000000002,"5min":7.0039999999999996,"15min":7.0039999999999996},"max":{"1min":20.760999999999999,"5min":29.818999999999999,"15min":29.818999999999999},"last":8.2870000000000008}]},{"osd":5,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":14.279999999999999,"5min":16.236999999999998,"15min":16.689},"min":{"1min":7.8920000000000003,"5min":7.8920000000000003,"15min":7.8920000000000003},"max":{"1min":25.75,"5min":55.784999999999997,"15min":55.784999999999997},"last":9.5549999999999997},{"interface":"front","average":{"1min":18.994,"5min":18.856999999999999,"15min":19.475999999999999},"min":{"1min":11.208,"5min":9.5800000000000001,"15min":9.5800000000000001},"max":{"1min":68.004999999999995,"5min":68.004999999999995,"15min":68.004999999999995},"last":13.071999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":15.708,"5min":16.609000000000002,"15min":17.382000000000001},"min":{"1min":10.180999999999999,"5min":9.1929999999999996,"15min":9.1929999999999996},"max":{"1min":27.353999999999999,"5min":58.201999999999998,"15min":58.201999999999998},"last":18.898},{"interface":"front","average":{"1min":16.324999999999999,"5min":16.748000000000001,"15min":17.001000000000001},"min":{"1min":9.6980000000000004,"5min":8.8390000000000004,"15min":8.8390000000000004},"max":{"1min":26.257999999999999,"5min":37.253,"15min":37.253},"last":12.787000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":21.238,"5min":18.891999999999999,"15min":19.809999999999999},"min":{"1min":8.9770000000000003,"5min":8.2050000000000001,"15min":8.2050000000000001},"max":{"1min":68.950000000000003,"5min":68.950000000000003,"15min":68.950000000000003},"last":13.592000000000001},{"interface":"front","average":{"1min":18.027999999999999,"5min":18.108000000000001,"15min":19.058},"min":{"1min":8.5850000000000009,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":31.367000000000001,"5min":59.079000000000001,"15min":59.079000000000001},"last":17.036000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313277,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27560,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344280,"statfs":{"total":96636764160,"available":96608542720,"internally_reserved":0,"allocated":552960,"data_stored":277024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":8.4659999999999993,"5min":9.2919999999999998,"15min":9.0730000000000004},"min":{"1min":4.8079999999999998,"5min":4.7169999999999996,"15min":4.7169999999999996},"max":{"1min":16.690000000000001,"5min":30.600999999999999,"15min":30.600999999999999},"last":7.4450000000000003},{"interface":"front","average":{"1min":10.06,"5min":9.4169999999999998,"15min":9.2210000000000001},"min":{"1min":4.9779999999999998,"5min":4.5430000000000001,"15min":4.5430000000000001},"max":{"1min":26.928000000000001,"5min":28.077000000000002,"15min":28.077000000000002},"last":6.9459999999999997}]},{"osd":1,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":11.928000000000001,"5min":12.59,"15min":11.42},"min":{"1min":5.7270000000000003,"5min":5.7270000000000003,"15min":5.7270000000000003},"max":{"1min":23.419,"5min":83.581000000000003,"15min":83.581000000000003},"last":15.584},{"interface":"front","average":{"1min":11.988,"5min":12.587999999999999,"15min":12.467000000000001},"min":{"1min":6.3719999999999999,"5min":6.3719999999999999,"15min":6.3719999999999999},"max":{"1min":27.940000000000001,"5min":81.722999999999999,"15min":81.722999999999999},"last":9.0370000000000008}]},{"osd":2,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.329000000000001,"5min":13.84,"15min":13.672000000000001},"min":{"1min":6.6559999999999997,"5min":6.5970000000000004,"15min":6.5970000000000004},"max":{"1min":42.506,"5min":42.506,"15min":42.506},"last":28.068999999999999},{"interface":"front","average":{"1min":15.164,"5min":15.433,"15min":16.823},"min":{"1min":7.673,"5min":6.9800000000000004,"15min":6.9800000000000004},"max":{"1min":43.484999999999999,"5min":43.484999999999999,"15min":43.484999999999999},"last":22.141999999999999}]},{"osd":3,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":15.202999999999999,"5min":16.597999999999999,"15min":16.596},"min":{"1min":10.531000000000001,"5min":8.8420000000000005,"15min":8.8420000000000005},"max":{"1min":35.677999999999997,"5min":65.739999999999995,"15min":65.739999999999995},"last":20.928000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":13.198,"15min":12.842000000000001},"min":{"1min":6.2119999999999997,"5min":6.2119999999999997,"15min":6.2119999999999997},"max":{"1min":36.258000000000003,"5min":64.980000000000004,"15min":64.980000000000004},"last":16.460999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":14.795,"5min":13.742000000000001,"15min":12.800000000000001},"min":{"1min":8.3670000000000009,"5min":7.0339999999999998,"15min":7.0339999999999998},"max":{"1min":24.486999999999998,"5min":37.746000000000002,"15min":37.746000000000002},"last":27.172000000000001},{"interface":"front","average":{"1min":14.532999999999999,"5min":16.396999999999998,"15min":17.324999999999999},"min":{"1min":10.377000000000001,"5min":9.2070000000000007,"15min":9.2070000000000007},"max":{"1min":22.492000000000001,"5min":37.393000000000001,"15min":37.393000000000001},"last":29.061}]},{"osd":5,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":12.417999999999999,"5min":11.829000000000001,"15min":11.496},"min":{"1min":7.6440000000000001,"5min":6.2309999999999999,"15min":6.2309999999999999},"max":{"1min":23.266999999999999,"5min":29.754999999999999,"15min":29.754999999999999},"last":8.4179999999999993},{"interface":"front","average":{"1min":15.071,"5min":16.338999999999999,"15min":16.789999999999999},"min":{"1min":8.407,"5min":8.407,"15min":8.407},"max":{"1min":24.846,"5min":38.618000000000002,"15min":38.618000000000002},"last":23.434000000000001}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.326000000000001,"5min":13.771000000000001,"15min":13.208},"min":{"1min":8.0060000000000002,"5min":6.3230000000000004,"15min":6.3230000000000004},"max":{"1min":19.231000000000002,"5min":42.829999999999998,"15min":42.829999999999998},"last":11.368},{"interface":"front","average":{"1min":15.842000000000001,"5min":17.606000000000002,"15min":18.088999999999999},"min":{"1min":11.712,"5min":10.749000000000001,"15min":10.749000000000001},"max":{"1min":22.695,"5min":44.491999999999997,"15min":44.491999999999997},"last":14.314}]}]},{"osd":6,"up_from":0,"seq":90194313274,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":21,"seq":90194313276,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":557056,"data_stored":277043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":10.545,"5min":10.287000000000001,"15min":11.180999999999999},"min":{"1min":5.0599999999999996,"5min":4.29,"15min":4.29},"max":{"1min":27.635000000000002,"5min":50.500999999999998,"15min":50.500999999999998},"last":6.4889999999999999},{"interface":"front","average":{"1min":11.278,"5min":12.206,"15min":14.045999999999999},"min":{"1min":5.9470000000000001,"5min":5.1360000000000001,"15min":5.1360000000000001},"max":{"1min":36.720999999999997,"5min":51.965000000000003,"15min":51.965000000000003},"last":8.9049999999999994}]},{"osd":1,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":14.144,"5min":15.917,"15min":18.513999999999999},"min":{"1min":8.2029999999999994,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":31.433,"5min":137.56700000000001,"15min":137.56700000000001},"last":10.946999999999999},{"interface":"front","average":{"1min":13.198,"5min":14.635,"15min":16.535},"min":{"1min":6.4779999999999998,"5min":5.0060000000000002,"15min":5.0060000000000002},"max":{"1min":24.753,"5min":136.321,"15min":136.321},"last":5.5720000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":12.728999999999999,"5min":14.698,"15min":16.170000000000002},"min":{"1min":7.2480000000000002,"5min":7.2480000000000002,"15min":7.2480000000000002},"max":{"1min":26.210000000000001,"5min":73.831000000000003,"15min":73.831000000000003},"last":9.7189999999999994},{"interface":"front","average":{"1min":10.778,"5min":13.547000000000001,"15min":15.545},"min":{"1min":6.7050000000000001,"5min":6.7050000000000001,"15min":6.7050000000000001},"max":{"1min":19.884,"5min":71.831999999999994,"15min":71.831999999999994},"last":8.1199999999999992}]},{"osd":4,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":14.756,"5min":21.741,"15min":28.48},"min":{"1min":7.7949999999999999,"5min":7.7949999999999999,"15min":7.7949999999999999},"max":{"1min":32.465000000000003,"5min":342.34500000000003,"15min":342.34500000000003},"last":11.999000000000001},{"interface":"front","average":{"1min":16.163,"5min":17.98,"15min":20.091000000000001},"min":{"1min":10.538,"5min":8.0299999999999994,"15min":8.0299999999999994},"max":{"1min":29.649999999999999,"5min":127.857,"15min":127.857},"last":11.210000000000001}]},{"osd":5,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.073,"5min":15.792999999999999,"15min":16},"min":{"1min":7.5250000000000004,"5min":7.2039999999999997,"15min":7.2039999999999997},"max":{"1min":55.005000000000003,"5min":58.715000000000003,"15min":58.715000000000003},"last":37.534999999999997},{"interface":"front","average":{"1min":19.274000000000001,"5min":18.774000000000001,"15min":19.256},"min":{"1min":7.5869999999999997,"5min":7.2370000000000001,"15min":7.2370000000000001},"max":{"1min":54.058,"5min":58.212000000000003,"15min":58.212000000000003},"last":38.450000000000003}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.295000000000002,"5min":23.492000000000001,"15min":30.053999999999998},"min":{"1min":11.096,"5min":8.6630000000000003,"15min":8.6630000000000003},"max":{"1min":25.298999999999999,"5min":388.96100000000001,"15min":388.96100000000001},"last":15.696},{"interface":"front","average":{"1min":13.961,"5min":24.18,"15min":32.835999999999999},"min":{"1min":9.4079999999999995,"5min":8.5519999999999996,"15min":8.5519999999999996},"max":{"1min":20.439,"5min":448.69200000000001,"15min":448.69200000000001},"last":13.44}]},{"osd":7,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":17.707999999999998,"5min":21.858000000000001,"15min":25.350999999999999},"min":{"1min":10.74,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":33.82,"5min":177.596,"15min":177.596},"last":14.037000000000001},{"interface":"front","average":{"1min":17.922000000000001,"5min":22.812000000000001,"15min":26.838000000000001},"min":{"1min":11.428000000000001,"5min":9.6270000000000007,"15min":9.6270000000000007},"max":{"1min":28.056999999999999,"5min":231.03800000000001,"15min":231.03800000000001},"last":12.877000000000001}]}]},{"osd":4,"up_from":0,"seq":85899345981,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":68719476795,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":15,"seq":64424509498,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":285662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":10.943,"5min":10.377000000000001,"15min":9.9009999999999998},"min":{"1min":5.3259999999999996,"5min":5.0019999999999998,"15min":5.0019999999999998},"max":{"1min":21.992999999999999,"5min":34.015000000000001,"15min":34.015000000000001},"last":10.34},{"interface":"front","average":{"1min":12.393000000000001,"5min":11.308999999999999,"15min":11.339},"min":{"1min":7.0629999999999997,"5min":4.7210000000000001,"15min":4.7210000000000001},"max":{"1min":29.138000000000002,"5min":36.636000000000003,"15min":36.636000000000003},"last":5.7190000000000003}]},{"osd":2,"last update":"Fri Aug 30 16:03:13 2024","interfaces":[{"interface":"back","average":{"1min":11.630000000000001,"5min":11.348000000000001,"15min":11.256},"min":{"1min":6.1109999999999998,"5min":3.3370000000000002,"15min":3.3370000000000002},"max":{"1min":21.43,"5min":46.768999999999998,"15min":46.768999999999998},"last":12.246},{"interface":"front","average":{"1min":10.179,"5min":13.541,"15min":14.135},"min":{"1min":7.5209999999999999,"5min":5.702,"15min":5.702},"max":{"1min":19.795000000000002,"5min":43.921999999999997,"15min":43.921999999999997},"last":9.1449999999999996}]},{"osd":3,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":14.497999999999999,"5min":18.655999999999999,"15min":23.094999999999999},"min":{"1min":7.4530000000000003,"5min":6.2839999999999998,"15min":6.2839999999999998},"max":{"1min":36.070999999999998,"5min":200.155,"15min":200.155},"last":7.3639999999999999},{"interface":"front","average":{"1min":18.696999999999999,"5min":23.181999999999999,"15min":28.526},"min":{"1min":11.661,"5min":7.0940000000000003,"15min":7.0940000000000003},"max":{"1min":37.436,"5min":267.03100000000001,"15min":267.03100000000001},"last":8.4339999999999993}]},{"osd":4,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":14.69,"5min":14.949999999999999,"15min":15.651999999999999},"min":{"1min":7.4329999999999998,"5min":6.6429999999999998,"15min":6.6429999999999998},"max":{"1min":31.611999999999998,"5min":56.200000000000003,"15min":56.200000000000003},"last":6.6150000000000002},{"interface":"front","average":{"1min":17.827000000000002,"5min":16.105,"15min":15.862},"min":{"1min":11.4,"5min":8.609,"15min":8.609},"max":{"1min":46.875999999999998,"5min":48.534999999999997,"15min":48.534999999999997},"last":13.257999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:03:13 2024","interfaces":[{"interface":"back","average":{"1min":15.835000000000001,"5min":15.973000000000001,"15min":16.18},"min":{"1min":11.196,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":35.521000000000001,"5min":35.521000000000001,"15min":35.521000000000001},"last":13.808},{"interface":"front","average":{"1min":17.591000000000001,"5min":17.614000000000001,"15min":17.827999999999999},"min":{"1min":9.2880000000000003,"5min":4.75,"15min":4.75},"max":{"1min":36.832000000000001,"5min":49.396000000000001,"15min":49.396000000000001},"last":28.094999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":13.808999999999999,"5min":14.292,"15min":14.385999999999999},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":39.460999999999999,"5min":59.328000000000003,"15min":59.328000000000003},"last":23.652999999999999},{"interface":"front","average":{"1min":19.225000000000001,"5min":18.331,"15min":18.533999999999999},"min":{"1min":10.781000000000001,"5min":10.005000000000001,"15min":10.005000000000001},"max":{"1min":46.237000000000002,"5min":58.567,"15min":58.567},"last":38.289000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":22.013000000000002,"5min":20.611000000000001,"15min":21.581},"min":{"1min":14.118,"5min":9.1180000000000003,"15min":9.1180000000000003},"max":{"1min":42.585000000000001,"5min":59.515999999999998,"15min":59.515999999999998},"last":25.981999999999999},{"interface":"front","average":{"1min":13.846,"5min":14.859999999999999,"15min":15.468},"min":{"1min":9.3550000000000004,"5min":7.1619999999999999,"15min":7.1619999999999999},"max":{"1min":27.023,"5min":58.807000000000002,"15min":58.807000000000002},"last":10.904999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":606208,"data_stored":683502,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:03:17 2024","interfaces":[{"interface":"back","average":{"1min":10.364000000000001,"5min":11.343,"15min":13.547000000000001},"min":{"1min":4.1299999999999999,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":53.558999999999997,"5min":111.313,"15min":111.313},"last":7.6630000000000003},{"interface":"front","average":{"1min":11.138999999999999,"5min":12.304,"15min":14.882},"min":{"1min":5.4809999999999999,"5min":4.7750000000000004,"15min":4.7750000000000004},"max":{"1min":52.691000000000003,"5min":107.937,"15min":107.937},"last":11}]},{"osd":1,"last update":"Fri Aug 30 16:03:29 2024","interfaces":[{"interface":"back","average":{"1min":13.371,"5min":12.039,"15min":13.057},"min":{"1min":5.8719999999999999,"5min":4.657,"15min":4.657},"max":{"1min":34.545999999999999,"5min":46.988,"15min":46.988},"last":8.5739999999999998},{"interface":"front","average":{"1min":15.677,"5min":14.946,"15min":15.869999999999999},"min":{"1min":6.7350000000000003,"5min":6.7350000000000003,"15min":6.7350000000000003},"max":{"1min":33.203000000000003,"5min":47.661000000000001,"15min":47.661000000000001},"last":22.945}]},{"osd":2,"last update":"Fri Aug 30 16:03:17 2024","interfaces":[{"interface":"back","average":{"1min":16.106000000000002,"5min":17.030000000000001,"15min":18.556000000000001},"min":{"1min":9.6470000000000002,"5min":4.7220000000000004,"15min":4.7220000000000004},"max":{"1min":41.549999999999997,"5min":84.317999999999998,"15min":84.317999999999998},"last":27.286999999999999},{"interface":"front","average":{"1min":14.890000000000001,"5min":15.475,"15min":16.731000000000002},"min":{"1min":6.9530000000000003,"5min":6.0279999999999996,"15min":6.0279999999999996},"max":{"1min":43.314999999999998,"5min":99.884,"15min":99.884},"last":21.169}]},{"osd":3,"last update":"Fri Aug 30 16:03:26 2024","interfaces":[{"interface":"back","average":{"1min":15.500999999999999,"5min":17.43,"15min":17.207999999999998},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":6.2149999999999999},"max":{"1min":28.585000000000001,"5min":117.346,"15min":117.346},"last":24.276},{"interface":"front","average":{"1min":13.127000000000001,"5min":16.134,"15min":16.379999999999999},"min":{"1min":7.1719999999999997,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":20.510000000000002,"5min":119.23099999999999,"15min":119.23099999999999},"last":9.9469999999999992}]},{"osd":4,"last update":"Fri Aug 30 16:03:26 2024","interfaces":[{"interface":"back","average":{"1min":16.254000000000001,"5min":17.491,"15min":17.895},"min":{"1min":7.9870000000000001,"5min":7.9870000000000001,"15min":7.9870000000000001},"max":{"1min":28.026,"5min":38.095999999999997,"15min":38.095999999999997},"last":25.402999999999999},{"interface":"front","average":{"1min":14.409000000000001,"5min":16.515000000000001,"15min":17.207999999999998},"min":{"1min":7.5030000000000001,"5min":7.5030000000000001,"15min":7.5030000000000001},"max":{"1min":31.367000000000001,"5min":40.649000000000001,"15min":40.649000000000001},"last":12.347}]},{"osd":6,"last update":"Fri Aug 30 16:03:26 2024","interfaces":[{"interface":"back","average":{"1min":9.516,"5min":10.686,"15min":11.025},"min":{"1min":6.7809999999999997,"5min":6.367,"15min":6.367},"max":{"1min":15.27,"5min":26.344999999999999,"15min":26.344999999999999},"last":8.5960000000000001},{"interface":"front","average":{"1min":19.596,"5min":17.696999999999999,"15min":17.396000000000001},"min":{"1min":11.433,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":58.052999999999997,"5min":58.052999999999997,"15min":58.052999999999997},"last":13.423}]},{"osd":7,"last update":"Fri Aug 30 16:03:26 2024","interfaces":[{"interface":"back","average":{"1min":14.516999999999999,"5min":16.550999999999998,"15min":17.843},"min":{"1min":10.054,"5min":9.0909999999999993,"15min":9.0909999999999993},"max":{"1min":26.913,"5min":43.604999999999997,"15min":43.604999999999997},"last":19.099},{"interface":"front","average":{"1min":11.013999999999999,"5min":14.605,"15min":17.475999999999999},"min":{"1min":6.3810000000000002,"5min":6.3810000000000002,"15min":6.3810000000000002},"max":{"1min":15.276,"5min":88.278000000000006,"15min":88.278000000000006},"last":9.2629999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:03:49.408 INFO:tasks.thrashosds.thrasher:Setting 2.3 to [7, 0] 2024-08-30T16:03:49.408 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.3', '7', '0'] 2024-08-30T16:03:49.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.3 7 0 2024-08-30T16:03:49.419 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:10.956 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:03:49.449 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:49.448+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:49.488 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:49.501 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:49.555 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:49.553+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:49.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:03:49.629 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:49.629 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:49.638 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:49.651 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:49.649+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:49.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:03:49.752 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:49.751+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:49.769 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:49.769 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:49.778 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:49.853 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:49.851+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:49.925 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:49.925 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:49.936 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:49.979 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:49.976+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:50.055 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:50.053+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:50.112 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:50.112 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:50.121 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:50.166 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:50.164+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:50.247 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:50.248 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:50.257 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:50.257 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:50.254+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:50.300 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:50.300 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:50.357 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:50.355+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:50.420 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:50.420 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:50.429 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:50.433 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:50.433 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:50.458 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:50.456+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:50.526 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:50.526 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:50.555 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:50.555 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:50.558 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:50.557+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:50.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:50.640 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:50.641 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:50.659 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:50.658+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:50.696 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:50.696 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:50.696 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:50.696 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:50.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:50.742 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:50.742 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:50.785 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:50.784+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:50.824 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:50.824 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:50.861 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:50.860+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:50.870 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:50.870 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:50.871 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:50.871 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:50.871 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:50.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:50.890 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:03:50.962 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:50.960+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:50.980 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:50.980 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:50.985 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:03:50.986 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:03:51.053 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:03:51.053 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:51.062 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:51.061+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:51.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:51.078 INFO:teuthology.orchestra.run.smithi050.stderr:set 2.3 pg_upmap mapping to [7,0] 2024-08-30T16:03:51.147 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-08-30T16:03:51.152 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:03:51.164 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:51.162+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:51.168 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-08-30T16:03:51.168 INFO:tasks.mon_recovery:removing mon c 2024-08-30T16:03:51.168 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-30T16:03:51.169 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:03:51.247 INFO:tasks.ceph.mon.c.smithi050.stderr:2024-08-30T16:03:51.233+0000 a084640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 42899) UID: 0 2024-08-30T16:03:51.249 INFO:tasks.ceph.mon.c.smithi050.stderr:2024-08-30T16:03:51.235+0000 a084640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-08-30T16:03:51.264 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:51.263+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:51.307 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:51.307 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:51.308 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:51.308 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:51.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:51.365 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:51.364+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:51.466 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:51.465+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:51.503 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:51.504 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:51.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:51.568 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:51.566+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:51.673 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:51.672+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:51.748 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:51.748 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:51.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:51.749 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:51.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.749 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:51.749 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:51.749 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 270.47403785799997, 2024-08-30T16:03:51.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:03:51.749 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:51.749 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:51.749 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:51.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.750 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:51.750 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:51.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:51.750 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.750 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:51.750 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:51.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:51.750 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.750 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:51.750 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:03:51.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:03:51.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:51.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:03:51.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:03:51.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:51.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:03:51.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:03:51.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:51.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:03:51.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:03:51.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:51.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:03:51.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:03:51.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:51.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:03:51.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:03:51.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:51.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:03:51.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:03:51.756 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:51.756 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:51.756 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:51.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.757 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:51.757 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:51.757 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 270.201110497, 2024-08-30T16:03:51.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:03:51.757 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:51.757 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:51.757 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:51.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:51.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:51.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:51.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:51.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:51.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:51.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:51.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:03:51.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:03:51.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:51.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:03:51.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:03:51.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:51.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:03:51.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:03:51.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:51.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:03:51.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:03:51.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:51.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:03:51.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:03:51.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:51.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:03:51.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:03:51.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:51.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:51.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:51.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:03:51.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:03:51.761 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:51.761 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:51.761 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:51.761 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:51.761 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:51.761 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:51.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:51.803 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:51.802+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:51.870 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:51.868+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:51.939 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:51.939 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:51.939 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:51.939 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:51.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:51.989 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:51.988+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:52.072 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:52.071+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:52.116 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:52.116 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:52.116 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:52.116 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:52.117 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:52.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:52.173 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:52.171+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:52.274 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:52.272+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:52.299 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:52.299 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:52.299 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:52.299 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:52.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.299 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:03:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 266.92209666500003, 2024-08-30T16:03:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:03:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.300 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:03:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:03:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:03:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:03:52.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:03:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:03:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:03:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:03:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:52.302 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:03:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:03:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:03:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:03:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:03:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:03:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.304 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:52.304 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:03:52.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:03:52.304 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.304 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:52.304 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.304 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.304 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:52.304 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:52.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:52.393 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:52.391+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:52.476 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:52.474+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:52.477 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:52.477 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:52.478 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:52.478 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:52.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:52.577 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:52.575+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:52.655 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:52.656 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:52.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:52.709 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:52.707+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:52.781 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:52.780+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:52.822 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:52.822 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:52.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:52.822 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:52.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.822 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:52.822 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:52.822 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 268.36676298499998, 2024-08-30T16:03:52.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:03:52.822 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:52.822 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:52.823 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:52.823 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:52.823 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:52.823 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:52.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.823 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:52.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:52.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:52.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:52.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:52.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:52.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:03:52.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:03:52.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:52.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:03:52.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:03:52.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:52.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:03:52.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:03:52.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:52.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:03:52.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:03:52.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:52.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:03:52.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:03:52.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:03:52.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:03:52.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:03:52.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:52.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:03:52.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:03:52.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:52.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:03:52.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:03:52.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:03:52.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:03:52.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:03:52.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:52.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:03:52.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:03:52.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:52.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:03:52.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:03:52.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:03:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:03:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:03:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:03:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:03:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 267.25089665000002, 2024-08-30T16:03:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:03:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:52.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:52.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:52.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:03:52.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:52.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:52.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:03:52.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:03:52.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:52.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:03:52.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:03:52.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:52.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:03:52.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:03:52.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:52.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:03:52.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:03:52.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:52.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:03:52.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:03:52.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:52.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:03:52.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:03:52.836 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.836 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:52.836 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.837 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:52.837 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:52.837 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 267.19130515199998, 2024-08-30T16:03:52.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:03:52.837 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:52.837 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:52.837 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:52.837 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:52.838 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:52.838 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:03:52.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.838 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:52.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.838 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:52.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:52.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:52.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:52.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:52.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:03:52.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:03:52.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:52.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:03:52.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:03:52.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:52.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:03:52.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:03:52.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:52.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:03:52.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:03:52.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:52.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:03:52.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:03:52.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:52.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:03:52.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:03:52.841 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.841 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:52.841 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.842 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:03:52.842 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:52.842 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 267.16460646500002, 2024-08-30T16:03:52.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:03:52.842 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:52.843 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:52.843 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:52.843 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:52.843 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:52.843 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:52.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.843 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:52.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:52.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:03:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:52.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:03:52.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:03:52.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:52.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:03:52.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:03:52.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:52.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:03:52.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:03:52.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:52.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:03:52.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:03:52.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:52.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:03:52.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:03:52.847 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.847 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:52.847 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.848 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:52.848 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:52.848 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 266.29698993699998, 2024-08-30T16:03:52.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:03:52.848 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:52.848 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:52.848 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:52.848 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:52.848 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:52.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:03:52.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:52.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:52.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:03:52.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:03:52.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:52.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:03:52.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:03:52.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:52.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:03:52.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:03:52.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:52.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:03:52.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:03:52.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:52.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:03:52.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:03:52.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:52.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:03:52.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:03:52.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:52.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:03:52.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:03:52.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:52.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:03:52.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:03:52.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:52.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:03:52.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:03:52.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:52.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:03:52.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:03:52.854 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.855 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:52.855 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.855 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:03:52.855 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:52.855 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 266.13774285800002, 2024-08-30T16:03:52.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:03:52.855 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:52.855 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:52.855 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:52.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:52.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:52.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:52.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:52.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:52.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:03:52.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:03:52.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:52.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:03:52.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:03:52.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:52.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:03:52.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:03:52.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:52.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:03:52.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:03:52.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:52.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:03:52.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:03:52.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:03:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:03:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:03:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:03:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 265.86156879200001, 2024-08-30T16:03:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:03:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:52.861 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:52.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:52.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:52.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:52.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:52.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:03:52.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:52.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:52.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:03:52.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:03:52.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:52.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:03:52.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:03:52.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:52.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:03:52.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:03:52.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:52.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:03:52.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:03:52.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:52.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:03:52.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:03:52.867 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.867 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:52.867 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.867 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:52.867 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:52.867 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 265.20327721299998, 2024-08-30T16:03:52.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:03:52.868 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:52.868 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:52.868 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:52.868 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:52.868 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:52.868 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:03:52.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.868 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:52.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:52.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:52.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:52.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:52.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:52.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:03:52.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:52.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:52.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:03:52.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:03:52.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:52.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:03:52.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:03:52.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:52.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:03:52.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:03:52.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:52.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:03:52.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:03:52.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:52.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:03:52.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:03:52.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:52.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:03:52.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:03:52.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:52.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:03:52.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:03:52.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:52.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:03:52.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:03:52.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:52.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:03:52.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:03:52.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:52.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:03:52.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:03:52.874 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.875 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:52.875 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.875 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:03:52.875 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:52.875 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 265.10736392899997, 2024-08-30T16:03:52.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:03:52.875 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:52.876 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:52.876 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:52.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:52.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:52.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:52.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:52.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:52.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:03:52.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:52.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:52.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:03:52.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:03:52.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:52.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:03:52.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:03:52.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:52.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:03:52.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:03:52.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:52.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:03:52.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:03:52.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:52.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:03:52.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:03:52.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:52.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:03:52.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:03:52.881 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.882 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:52.882 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.882 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:52.882 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:52.882 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 205.856003418, 2024-08-30T16:03:52.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:03:52.883 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:52.883 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:52.883 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:52.883 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:52.883 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:52.883 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:52.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.883 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:52.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:52.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:52.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:52.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:52.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:52.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:03:52.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:03:52.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:52.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:03:52.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:03:52.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:52.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:03:52.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:03:52.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:52.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:03:52.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:03:52.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:52.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:03:52.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:03:52.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:52.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:03:52.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:03:52.887 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.887 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:52.888 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.888 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:03:52.888 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:52.888 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 145.84689873100001, 2024-08-30T16:03:52.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:03:52.888 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:52.888 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:52.888 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:52.888 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:52.889 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:52.889 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:52.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.889 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:52.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:52.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:52.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:52.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:52.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:52.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:52.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:03:52.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:03:52.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:52.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:03:52.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:03:52.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:52.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:03:52.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:03:52.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:52.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:03:52.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:03:52.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:52.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:03:52.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:03:52.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:52.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:52.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:52.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:03:52.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:03:52.893 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.894 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:52.894 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.894 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:52.894 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:52.894 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:52.896 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:52.897 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:52.894+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:52.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:03:53.022 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:53.022 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:53.022 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:53.023 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:53.031 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:53.029+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:53.032 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:53.098 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:53.096+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:53.158 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:53.158 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:53.158 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:53.158 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:53.158 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:53.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-08-30T16:03:53.168 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:53.212 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:53.210+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:53.227 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:53.227 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:53.294 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:53.294 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:53.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:53.294 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:03:53.294 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:53.304 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:53.321 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:53.321 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:53.324 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:53.323+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:53.400 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:53.399+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:53.426 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:53.426 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:53.466 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:53.466 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:53.466 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:53.474 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:53.501 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:53.500+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:53.530 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:53.530 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:53.610 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:53.610 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:53.610 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:53.611 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:53.611 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:53.619 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:53.630 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:53.630 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:53.630 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:53.628+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:53.703 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:53.702+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:53.714 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:53.714 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:53.716 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:03:53.754 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:53.754 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:53.764 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:53.808 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:03:53.808 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:03:53.810 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:53.808+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:03:53.812 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:03:53.890 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:53.890 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:53.899 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:53.905 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:53.904+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:54.007 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:54.005+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:54.064 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:54.064 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:54.073 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:54.108 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:54.107+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:54.209 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:54.207+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:54.240 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:54.240 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:54.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:54.310 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:54.308+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:54.379 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:54.379 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:54.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:54.430 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:54.429+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:54.512 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:54.510+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:54.556 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:54.556 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:54.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:54.612 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:54.610+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:54.713 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:54.711+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:54.732 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:54.732 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:54.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:54.814 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:54.813+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:54.868 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:54.869 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:54.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:54.915 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:54.913+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:55.016 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:55.014+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:55.045 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:55.045 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:55.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:55.116 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:55.115+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:55.217 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:55.216+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:55.222 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:55.222 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:55.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:55.319 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:55.317+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:55.400 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:55.401 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:55.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:55.419 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:55.418+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:55.520 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:55.519+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:55.539 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:55.539 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:55.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:55.621 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:55.619+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:55.715 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:55.715 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:55.722 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:55.721+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:55.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:55.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:03:55.856 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:55.856 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:55.857 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:55.855+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:55.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:55.923 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:55.922+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:56.025 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:56.023+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:56.045 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:56.045 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:56.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:56.062 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:03:56.062 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:03:56.098 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7] out_osds: [6, 4, 2] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-08-30T16:03:56.098 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:03:56.098 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-30T16:03:56.098 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-08-30T16:03:56.126 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:56.124+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:56.157 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:03:56.157 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:03:56.185 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:56.185 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:56.195 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:56.227 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:56.225+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:56.247 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:03:56.248 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:03:56.328 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:56.326+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:56.336 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:56.336 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:56.337 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:03:56.337 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:03:56.347 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:03:56.421 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:03:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:03:56.431 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:56.429+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:56.522 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:03:56.523 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:03:56.529 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:56.528+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:56.530 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:56.530 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:56.530 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:56.530 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:56.530 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:56.540 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:03:56.631 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:56.629+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:56.703 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:56.703 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:03:56.703 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:56.713 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:03:56.731 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:56.730+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:56.832 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:56.830+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:56.835 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:56.835 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:56.835 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:03:56.835 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:56.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:03:56.933 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:56.931+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:56.966 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:56.967 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:03:56.967 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:03:56.967 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:03:56.967 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:56.976 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:03:57.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-08-30T16:03:57.061 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:57.059+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:57.135 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:57.133+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:57.188 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:03:57.188 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:03:57.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:03:57.188 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:03:57.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.188 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:57.188 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:57.188 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 286.327267097, 2024-08-30T16:03:57.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:03:57.189 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.189 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.189 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:03:57.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.189 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:57.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:03:57.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:03:57.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:03:57.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:03:57.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:03:57.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:03:57.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:03:57.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:03:57.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:03:57.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:03:57.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:03:57.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:03:57.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:03:57.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:03:57.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:03:57.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:03:57.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:03:57.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:03:57.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:03:57.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:03:57.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:03:57.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:03:57.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:03:57.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:03:57.194 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.194 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.194 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.195 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:57.195 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:57.195 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 285.41085107700002, 2024-08-30T16:03:57.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:03:57.195 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.195 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.195 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.195 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.195 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.195 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:03:57.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.195 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:57.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:03:57.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:03:57.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:57.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:03:57.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:03:57.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:03:57.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:03:57.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:03:57.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:03:57.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:03:57.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:03:57.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:03:57.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:03:57.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:03:57.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:03:57.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:03:57.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:03:57.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:03:57.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:03:57.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.200 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.200 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:03:57.200 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:57.200 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 285.40914652399999, 2024-08-30T16:03:57.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:03:57.200 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.200 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.200 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.200 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:03:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:03:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:03:57.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:03:57.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:03:57.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:03:57.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:03:57.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:03:57.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:03:57.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:03:57.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:03:57.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:03:57.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:03:57.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:03:57.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:03:57.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:03:57.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:03:57.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:03:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:03:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:03:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 284.98969271700003, 2024-08-30T16:03:57.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:03:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:03:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:03:57.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:03:57.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:03:57.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:03:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:03:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:03:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:03:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:03:57.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:03:57.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:03:57.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:03:57.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:03:57.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:03:57.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:03:57.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:03:57.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:03:57.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:03:57.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:03:57.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:03:57.210 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.211 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.211 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.211 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.211 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:57.211 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 284.972804543, 2024-08-30T16:03:57.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:03:57.211 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.211 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.211 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.212 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:03:57.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.212 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:57.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:03:57.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:03:57.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:03:57.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:03:57.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:03:57.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.213 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:03:57.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:03:57.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:03:57.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:03:57.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:03:57.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:03:57.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:03:57.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:03:57.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:03:57.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:03:57.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:03:57.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:03:57.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:03:57.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:03:57.216 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.216 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.216 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.216 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.216 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:57.216 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 284.95980447400001, 2024-08-30T16:03:57.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:03:57.216 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.217 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.217 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.217 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.217 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.217 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:03:57.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.217 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:57.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:03:57.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:03:57.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:03:57.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:03:57.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:03:57.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:03:57.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:03:57.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:03:57.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:03:57.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:03:57.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:03:57.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:03:57.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:03:57.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:03:57.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:03:57.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:03:57.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:03:57.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:03:57.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:03:57.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.221 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.221 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.222 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.222 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:57.222 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 284.95731218200001, 2024-08-30T16:03:57.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:03:57.222 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.222 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.223 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.223 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.223 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.223 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:03:57.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.223 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:57.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:03:57.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:03:57.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:57.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:03:57.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:03:57.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:03:57.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:03:57.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:03:57.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:03:57.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:03:57.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:03:57.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:03:57.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:03:57.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:03:57.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:03:57.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:03:57.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:03:57.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:03:57.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:03:57.227 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.227 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.227 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.228 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.228 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:57.228 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 284.95655473400001, 2024-08-30T16:03:57.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:03:57.228 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.228 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.228 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.228 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.228 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.228 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:03:57.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.228 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:03:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:03:57.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:03:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:03:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:03:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:03:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:03:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.230 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:03:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:03:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:03:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:03:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:03:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:03:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:03:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:03:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:03:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:03:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:03:57.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:03:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 284.95582228400002, 2024-08-30T16:03:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:03:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.233 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:03:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:03:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:03:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:03:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:03:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:03:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:03:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:03:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:03:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:03:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:03:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:03:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:03:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:03:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:03:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:03:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:03:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:03:57.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:03:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:03:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 284.95328348499999, 2024-08-30T16:03:57.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:03:57.239 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.239 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.239 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:03:57.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.239 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:57.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:03:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:03:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:03:57.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:03:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:03:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:03:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:03:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:03:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:03:57.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:03:57.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:03:57.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:03:57.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:03:57.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:03:57.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:03:57.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:03:57.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:03:57.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:03:57.243 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.243 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.243 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.244 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.244 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:57.244 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 284.95253548400001, 2024-08-30T16:03:57.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:03:57.244 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.244 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.244 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.244 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.244 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.244 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:03:57.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.244 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:57.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:03:57.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:03:57.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:03:57.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:03:57.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:03:57.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:03:57.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:03:57.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:03:57.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:03:57.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:03:57.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:03:57.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:03:57.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:03:57.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:03:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:03:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:03:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:03:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:03:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:03:57.248 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.249 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.249 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.249 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.249 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:57.249 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 284.18041901200002, 2024-08-30T16:03:57.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:03:57.249 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.249 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.249 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.249 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.249 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.250 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:03:57.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.250 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:57.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:03:57.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:03:57.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:03:57.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:03:57.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:03:57.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:03:57.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:03:57.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:03:57.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:03:57.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:03:57.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:03:57.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:03:57.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:03:57.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:03:57.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:03:57.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:03:57.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:03:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:03:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:03:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 284.17836186599999, 2024-08-30T16:03:57.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:03:57.255 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.255 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.255 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.255 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.255 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.255 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:03:57.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.255 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:57.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:03:57.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:03:57.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:57.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:03:57.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:03:57.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:03:57.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:03:57.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:03:57.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:03:57.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:03:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:03:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:03:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:03:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:03:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:03:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:03:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:03:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:03:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:03:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.260 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:57.260 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 283.91541736599999, 2024-08-30T16:03:57.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:03:57.260 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.260 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.260 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.260 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.260 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.260 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:03:57.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.260 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:57.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:03:57.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:03:57.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:03:57.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:03:57.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:03:57.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:03:57.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:03:57.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:03:57.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:03:57.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:03:57.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:03:57.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:03:57.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:03:57.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:03:57.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:03:57.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:03:57.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:03:57.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:03:57.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:03:57.265 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.265 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.265 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.265 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.265 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:57.265 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 283.91432586299999, 2024-08-30T16:03:57.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:03:57.265 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.265 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.265 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.265 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.266 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.266 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:03:57.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.266 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:57.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:03:57.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:03:57.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:03:57.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:03:57.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:03:57.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:03:57.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:03:57.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:03:57.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:03:57.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:03:57.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:03:57.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:03:57.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:03:57.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:03:57.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:03:57.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:03:57.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:03:57.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:03:57.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:03:57.270 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.270 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.270 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.270 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.271 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:57.271 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 283.91357627799999, 2024-08-30T16:03:57.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:03:57.271 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.271 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.271 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.271 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.271 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.271 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:03:57.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.271 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:57.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:03:57.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:03:57.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:57.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:03:57.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:03:57.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:03:57.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:03:57.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:03:57.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:03:57.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:03:57.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:03:57.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:03:57.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:03:57.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:03:57.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:03:57.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:03:57.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:03:57.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:03:57.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:03:57.275 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.275 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.276 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.276 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.276 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:57.276 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 283.879844746, 2024-08-30T16:03:57.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:03:57.276 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.276 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.276 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.276 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.276 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.276 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:03:57.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.277 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:57.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:03:57.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:03:57.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:03:57.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:03:57.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:03:57.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:03:57.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:03:57.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:03:57.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:03:57.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:03:57.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:03:57.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:03:57.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:03:57.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:03:57.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:03:57.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:03:57.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:03:57.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:03:57.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:03:57.281 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.281 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.281 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.281 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.281 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:57.281 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 283.87931873799999, 2024-08-30T16:03:57.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:03:57.281 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.282 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.282 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.282 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.282 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.282 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:03:57.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.282 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:57.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:03:57.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:03:57.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:57.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:03:57.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:03:57.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:03:57.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:03:57.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:03:57.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:03:57.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:03:57.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:03:57.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:03:57.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:03:57.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:03:57.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:03:57.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:03:57.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:03:57.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:03:57.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:03:57.289 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.289 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.289 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.290 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:03:57.290 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:57.290 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 283.87886317499999, 2024-08-30T16:03:57.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:03:57.290 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.290 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.290 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.290 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.290 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.290 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:03:57.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.291 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:57.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:03:57.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:03:57.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:03:57.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:03:57.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:03:57.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:03:57.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:03:57.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:03:57.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:03:57.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:03:57.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:03:57.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:03:57.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:03:57.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:03:57.295 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.295 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.295 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.295 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:03:57.295 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:57.295 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 279.20834785900001, 2024-08-30T16:03:57.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:03:57.295 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:03:57.295 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:57.295 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:03:57.296 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:03:57.296 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:03:57.296 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:03:57.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.296 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:03:57.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:03:57.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:57.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:03:57.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:03:57.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:03:57.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:03:57.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:03:57.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:03:57.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:03:57.297 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:03:57.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:03:57.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:03:57.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:03:57.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:03:57.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:03:57.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:03:57.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:03:57.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:03:57.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:03:57.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:03:57.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:03:57.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:03:57.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:03:57.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:03:57.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:03:57.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:03:57.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:03:57.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:03:57.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:03:57.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:03:57.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:03:57.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:03:57.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:03:57.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:03:57.300 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.300 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.300 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.300 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:03:57.300 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:03:57.300 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:03:57.301 INFO:tasks.ceph.mon.c:Stopped 2024-08-30T16:03:57.301 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2024-08-30T16:03:57.301 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:03:57.302 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:03:57.303 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:57.301+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:57.407 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:57.405+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:57.473 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:57.473 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:57.483 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:03:57.525 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:57.523+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:57.604 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:57.603+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:57.647 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:57.647 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:57.657 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:03:57.706 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:57.705+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:57.781 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:03:57.781 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:03:57.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:03:57.835 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:57.833+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:57.908 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:57.907+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:57.929 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:57.930 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:57.930 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:57.930 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:57.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:03:58.023 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:58.022+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:58.108 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:58.108 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:58.108 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:58.108 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:58.108 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:58.111 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:58.109+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:58.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:03:58.212 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:58.211+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:58.285 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:58.285 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:58.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:58.285 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:03:58.285 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:58.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:03:58.313 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:58.312+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:58.415 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:03:58.413+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:03:58.422 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:58.422 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:58.422 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:58.422 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:58.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:03:58.515 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:58.514+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:58.558 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:58.559 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:58.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:03:58.651 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:58.650+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:58.716 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:03:58.714+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:03:58.739 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 277.46506185599998, 2024-08-30T16:03:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:03:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:03:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:03:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:03:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:03:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:03:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:03:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:03:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:03:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:03:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:03:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:03:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:03:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:03:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:03:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:03:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:03:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 277.192134495, 2024-08-30T16:03:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:03:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:03:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:03:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:03:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:03:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:03:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:03:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:03:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:03:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:03:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:03:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:03:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:03:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:03:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:03:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:03:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:58.749 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:58.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:03:58.834 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:58.833+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:58.918 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:58.917+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:58.920 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:58.920 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:58.920 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:58.920 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:58.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:03:59.019 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:59.018+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:59.099 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:59.099 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:59.099 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:59.099 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:59.099 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:59.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:03:59.153 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:59.151+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:59.221 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:59.219+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:59.241 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:59.241 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:59.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:59.241 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:59.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.241 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:03:59.242 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:59.242 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 273.864149376, 2024-08-30T16:03:59.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:03:59.242 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:59.242 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:59.242 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:59.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:59.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:59.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:59.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:03:59.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:59.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:03:59.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:03:59.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:59.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:03:59.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:03:59.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:59.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:03:59.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:03:59.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:59.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:03:59.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:03:59.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:59.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:03:59.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:03:59.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:59.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:03:59.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:03:59.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:59.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:03:59.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:03:59.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:59.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:03:59.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:03:59.246 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.246 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:59.246 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.246 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.246 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:59.247 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:59.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:03:59.334 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:59.333+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:59.418 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:59.418 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:59.419 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:03:59.419 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:59.422 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:03:59.421+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:03:59.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:03:59.523 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:59.522+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:03:59.596 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:03:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:03:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:03:59.596 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:59.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:03:59.645 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:03:59.644+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:03:59.725 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:03:59.724+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:03:59.797 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:03:59.797 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:03:59.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:03:59.797 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:03:59.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.797 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:59.798 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:59.798 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 275.34404603500002, 2024-08-30T16:03:59.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:03:59.798 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:59.798 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:59.798 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:59.798 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:59.798 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:59.798 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:59.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.798 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:59.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:59.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:59.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:59.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:03:59.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:59.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:03:59.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:03:59.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:59.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:03:59.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:03:59.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:59.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:03:59.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:03:59.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.801 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:59.801 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:03:59.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:03:59.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.801 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:59.801 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:03:59.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:03:59.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:03:59.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:03:59.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:03:59.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:59.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:03:59.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:03:59.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:59.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:03:59.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:03:59.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.803 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:03:59.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:03:59.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:03:59.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.803 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:59.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:03:59.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:03:59.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:59.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:03:59.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:03:59.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:59.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:03:59.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:03:59.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.805 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:59.805 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:03:59.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:03:59.805 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.805 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:59.805 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.805 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:03:59.805 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:59.805 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 274.2281797, 2024-08-30T16:03:59.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:03:59.805 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:59.806 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:59.806 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:59.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:59.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:59.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.807 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:59.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:03:59.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.807 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:59.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:03:59.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:59.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.807 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:59.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:03:59.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:03:59.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:59.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:03:59.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:03:59.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:59.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:03:59.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:03:59.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:59.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:03:59.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:03:59.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:59.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:03:59.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:03:59.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:59.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:03:59.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:03:59.810 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.810 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:59.810 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.810 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:03:59.810 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:59.810 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 274.16858820200002, 2024-08-30T16:03:59.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:03:59.810 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:59.810 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:59.810 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:59.810 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:59.811 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:59.811 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:03:59.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.811 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:59.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:59.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:59.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:59.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:03:59.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:59.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:03:59.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:03:59.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:59.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:03:59.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:03:59.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:59.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:03:59.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:03:59.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:59.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:03:59.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:03:59.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:59.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:03:59.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:03:59.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:59.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:03:59.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:03:59.815 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.815 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:59.815 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.815 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:03:59.815 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:59.816 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 274.141889515, 2024-08-30T16:03:59.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:03:59.816 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:59.816 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:59.816 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:59.816 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:59.816 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:59.816 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:59.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.817 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:59.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:59.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:59.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:59.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:03:59.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:59.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:03:59.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:03:59.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:59.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:03:59.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:03:59.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:59.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:03:59.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:03:59.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:59.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:03:59.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:03:59.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:59.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:03:59.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:03:59.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:59.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:03:59.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:03:59.822 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.822 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:59.822 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.823 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:59.823 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:59.823 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 273.27427298700002, 2024-08-30T16:03:59.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:03:59.823 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:59.823 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:59.823 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:59.823 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:59.823 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:59.823 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:59.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.824 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:59.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:59.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:59.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:59.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:03:59.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:59.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:03:59.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:03:59.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:59.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:03:59.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:03:59.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:59.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:03:59.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:03:59.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:59.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:03:59.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:03:59.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:59.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:03:59.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:03:59.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:59.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:03:59.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:03:59.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:59.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:03:59.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:03:59.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:59.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:03:59.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:03:59.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:59.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:03:59.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:03:59.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:59.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:03:59.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:03:59.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:59.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:03:59.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:03:59.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.830 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:59.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.830 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:03:59.830 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:59.830 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 273.11502590800001, 2024-08-30T16:03:59.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:03:59.831 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:59.831 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:59.831 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:59.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:59.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:59.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:59.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:03:59.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:59.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:03:59.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:03:59.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:59.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:03:59.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:03:59.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:59.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:03:59.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:03:59.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:59.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:03:59.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:03:59.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:59.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:03:59.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:03:59.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:59.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:03:59.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:03:59.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:59.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:03:59.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:03:59.835 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.835 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:59.835 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.835 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:03:59.835 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:59.836 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 272.838851842, 2024-08-30T16:03:59.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:03:59.836 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:59.836 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:59.836 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:59.836 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:59.836 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:59.836 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:03:59.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.836 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:59.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:59.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:59.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:59.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:03:59.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:59.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:03:59.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:03:59.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.838 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:59.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:03:59.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:03:59.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.838 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:59.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:03:59.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:03:59.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:59.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:03:59.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:03:59.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:59.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:03:59.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:03:59.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:59.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:03:59.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:03:59.840 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.840 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:59.840 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.840 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:59.840 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:59.840 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 272.18056026300002, 2024-08-30T16:03:59.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:03:59.840 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:59.840 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:03:59.840 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:59.841 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:03:59.841 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:03:59.841 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:03:59.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.841 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:59.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:59.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:59.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:59.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:03:59.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:59.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:03:59.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:59.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:59.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:03:59.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:03:59.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:59.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:03:59.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:03:59.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:59.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:03:59.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:03:59.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:59.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:03:59.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:03:59.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:59.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:03:59.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:03:59.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:03:59.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:03:59.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:03:59.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:03:59.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:03:59.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:03:59.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:03:59.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:03:59.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:03:59.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:03:59.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:03:59.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:03:59.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:59.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:03:59.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:03:59.848 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.848 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:59.848 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.849 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:03:59.849 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:59.849 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 272.08464697900001, 2024-08-30T16:03:59.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:03:59.849 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:59.849 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:03:59.849 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:59.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:59.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:59.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:59.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:03:59.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:59.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:03:59.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:03:59.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:59.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:03:59.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:03:59.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:59.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:03:59.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:03:59.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:59.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:03:59.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:03:59.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:59.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:03:59.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:03:59.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:03:59.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:03:59.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:03:59.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:59.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:03:59.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:03:59.853 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.853 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:59.853 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.853 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:03:59.853 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 212.83328646800001, 2024-08-30T16:03:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:03:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:03:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:59.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:03:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:03:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:03:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:59.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:03:59.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:03:59.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:59.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:03:59.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:03:59.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:59.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:03:59.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:03:59.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:03:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:03:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:03:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:03:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.857 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:03:59.858 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:59.858 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 152.82418178099999, 2024-08-30T16:03:59.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:03:59.858 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:03:59.858 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:03:59.858 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:03:59.858 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:03:59.858 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:03:59.858 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:03:59.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.858 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:03:59.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:03:59.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:03:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:03:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:03:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:03:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:03:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:03:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:03:59.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:03:59.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:03:59.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:03:59.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:03:59.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:03:59.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:03:59.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:03:59.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:03:59.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:03:59.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:03:59.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:03:59.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:03:59.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:03:59.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:03:59.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:03:59.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:03:59.861 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.861 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:59.861 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.861 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:03:59.861 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:03:59.861 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:03:59.863 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:03:59.863 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:03:59.862+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:03:59.964 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:03:59.963+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:00.003 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:00.003 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:00.003 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:00.013 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:00.065 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:00.064+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:00.136 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:00.136 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:00.137 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:00.137 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:00.137 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:00.145 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:00.166 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:00.165+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:00.267 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:00.266+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:00.310 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:00.310 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:04:00.310 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:00.320 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:00.369 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:00.367+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:00.446 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:00.446 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:00.446 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:00.446 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:00.455 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:00.470 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:00.469+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:00.571 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:00.569+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:00.579 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:00.579 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:00.579 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:00.579 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:00.580 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:00.589 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:00.672 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:00.670+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:00.760 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:00.760 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:00.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:00.760 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:00.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.760 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:00.760 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:00.761 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 289.89911012200002, 2024-08-30T16:04:00.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:04:00.761 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.761 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.761 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.761 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.761 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.761 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:04:00.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.761 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.761 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:00.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:00.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:04:00.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:04:00.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:04:00.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:04:00.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:04:00.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:04:00.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:04:00.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:04:00.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:04:00.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:04:00.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:04:00.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:04:00.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:04:00.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:04:00.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:04:00.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:04:00.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:04:00.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:04:00.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:04:00.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:04:00.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:04:00.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:04:00.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:04:00.770 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.770 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.770 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.770 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:00.770 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:00.771 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 288.98269410199998, 2024-08-30T16:04:00.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:04:00.771 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.771 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.771 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.771 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.771 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.771 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:04:00.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.771 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:00.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.772 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.772 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:00.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.772 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:04:00.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:00.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:04:00.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:04:00.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:04:00.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:04:00.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:04:00.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:04:00.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:04:00.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:04:00.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:04:00.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:04:00.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:04:00.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:04:00.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:04:00.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:04:00.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:04:00.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:04:00.776 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.776 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.776 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.777 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:00.777 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:00.777 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 288.98098954900001, 2024-08-30T16:04:00.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:04:00.777 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.777 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.777 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.777 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.777 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.777 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:04:00.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.778 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:00.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.778 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.778 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:00.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:04:00.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:04:00.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:04:00.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:04:00.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:04:00.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:04:00.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:04:00.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:04:00.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:04:00.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:04:00.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:04:00.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:04:00.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:04:00.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:04:00.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:04:00.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:04:00.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:04:00.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:04:00.786 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.786 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.786 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.786 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.787 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:00.787 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 288.56153574199999, 2024-08-30T16:04:00.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:04:00.787 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.787 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.788 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.788 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.788 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.788 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:04:00.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.789 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:00.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:00.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:04:00.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:04:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:04:00.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:04:00.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:04:00.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:04:00.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:04:00.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:04:00.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:04:00.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:04:00.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:04:00.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:04:00.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:04:00.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:04:00.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:04:00.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:04:00.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:04:00.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:04:00.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.799 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.800 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.800 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:00.800 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 288.54464756800002, 2024-08-30T16:04:00.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:04:00.801 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.801 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.801 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.801 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.802 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.802 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:04:00.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.802 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:00.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:00.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:04:00.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:04:00.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:04:00.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:04:00.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:04:00.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:04:00.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.807 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:04:00.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:04:00.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:04:00.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:04:00.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:04:00.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:04:00.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:04:00.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:04:00.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:04:00.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:04:00.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:04:00.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:04:00.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.814 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.814 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.814 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.814 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:00.814 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 288.53164749899997, 2024-08-30T16:04:00.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:04:00.814 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.814 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.814 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.814 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.814 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.815 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:04:00.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.815 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:00.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:00.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:04:00.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:04:00.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:04:00.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:04:00.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:04:00.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:04:00.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:04:00.817 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:04:00.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.817 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.817 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.817 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:04:00.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:04:00.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:04:00.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:04:00.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.818 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.818 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:04:00.818 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:04:00.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:04:00.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:04:00.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.819 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.819 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:04:00.819 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:04:00.819 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.819 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.819 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.820 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.820 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:00.820 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 288.52915520699997, 2024-08-30T16:04:00.820 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:04:00.820 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.820 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.820 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.820 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.820 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.820 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:04:00.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.821 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.821 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.821 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:00.821 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.821 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.821 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:00.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.822 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:04:00.822 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:00.822 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.822 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.822 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:04:00.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:04:00.823 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.823 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.823 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.823 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:04:00.823 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:04:00.824 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.824 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.824 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.824 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:04:00.824 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:04:00.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.825 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.825 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.825 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:04:00.825 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:04:00.825 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.826 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.826 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:04:00.826 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:04:00.826 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.826 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.827 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:04:00.827 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:04:00.827 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.827 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.827 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:04:00.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:04:00.828 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.828 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.828 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.828 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:04:00.828 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:04:00.829 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.829 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.829 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.829 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.829 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.829 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.830 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:00.830 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 288.52839775899997, 2024-08-30T16:04:00.830 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:04:00.830 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.830 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.830 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.830 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.831 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.831 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:04:00.831 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.831 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.831 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.831 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.832 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:00.832 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.832 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.832 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.832 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:00.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.833 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.833 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.833 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.833 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:04:00.833 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:04:00.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.834 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:04:00.834 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:04:00.834 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.834 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.834 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:04:00.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:04:00.835 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.835 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.835 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.835 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:04:00.835 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:04:00.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.836 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.836 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:04:00.836 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:04:00.836 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.836 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.837 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:04:00.837 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:04:00.837 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.837 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.837 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.838 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:04:00.838 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:04:00.838 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.838 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.838 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:04:00.839 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:04:00.839 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.839 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.839 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.839 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:04:00.840 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:04:00.840 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.840 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.840 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.840 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.841 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.841 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.841 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:00.841 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 288.52766530899999, 2024-08-30T16:04:00.841 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:04:00.841 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.841 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.842 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.842 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.842 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.842 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:04:00.842 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.842 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.842 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.842 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:00.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.843 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.843 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.843 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:00.843 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.843 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:04:00.844 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:00.844 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.844 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.844 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.844 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:04:00.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:04:00.845 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.845 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.845 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.845 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:04:00.845 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:04:00.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.846 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.846 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.846 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:04:00.846 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:04:00.846 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:04:00.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:04:00.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.847 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.847 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:04:00.847 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:04:00.847 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:04:00.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:04:00.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.848 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:04:00.848 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:04:00.848 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.848 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.848 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.849 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:04:00.849 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:04:00.849 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.849 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.849 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.849 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.849 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.850 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.850 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:00.850 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 288.52512651000001, 2024-08-30T16:04:00.850 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:04:00.850 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.850 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.850 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.850 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.850 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.850 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:04:00.850 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.851 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:00.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.851 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:00.851 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.851 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.851 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.851 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:04:00.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:00.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:04:00.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:04:00.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.852 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.852 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.852 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:04:00.852 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:04:00.852 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:04:00.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:04:00.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:04:00.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:04:00.853 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.853 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.853 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.853 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:04:00.853 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:04:00.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:04:00.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:04:00.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.854 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:04:00.854 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:04:00.854 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.854 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.854 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.855 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:04:00.855 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:04:00.855 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.855 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.855 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.855 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.855 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.855 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.855 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:00.855 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 288.52437850899997, 2024-08-30T16:04:00.856 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:04:00.856 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.856 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.856 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.856 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.856 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.856 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:04:00.856 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.856 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.856 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.856 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.856 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:00.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:00.857 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.857 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.857 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.857 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.857 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:04:00.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:04:00.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:04:00.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:04:00.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.858 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.858 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:04:00.858 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:04:00.858 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.858 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:04:00.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:04:00.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:04:00.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:04:00.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.859 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.859 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.859 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:04:00.859 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:04:00.859 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:04:00.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:04:00.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:04:00.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:04:00.860 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.860 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.860 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.860 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:04:00.860 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:04:00.861 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.861 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.861 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.861 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.861 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.861 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.861 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:00.861 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 287.75226203699998, 2024-08-30T16:04:00.861 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:04:00.861 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.861 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.861 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.861 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.862 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.862 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:04:00.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.862 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:00.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.862 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.862 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.862 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.862 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:00.862 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:04:00.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:00.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:04:00.863 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:04:00.863 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.863 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.863 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.863 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:04:00.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:04:00.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:04:00.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:04:00.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.864 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.864 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:04:00.864 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:04:00.864 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.864 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:04:00.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:04:00.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:04:00.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:04:00.865 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.865 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.865 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.865 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:04:00.865 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:04:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:04:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:04:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 287.75020489100001, 2024-08-30T16:04:00.866 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:04:00.867 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.867 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.867 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.867 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.867 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.867 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:04:00.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.867 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.867 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.867 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:00.867 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.867 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.867 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:00.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:04:00.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:00.868 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.868 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.868 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.868 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:04:00.868 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:04:00.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:04:00.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:04:00.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.869 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:04:00.869 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:04:00.869 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.869 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.869 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:04:00.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:04:00.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:04:00.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:04:00.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.870 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.870 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:04:00.870 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:04:00.870 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.870 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:04:00.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:04:00.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.871 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.871 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:04:00.871 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:04:00.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.871 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.871 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.871 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.871 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.871 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.872 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:00.872 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 287.48726039100001, 2024-08-30T16:04:00.872 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:04:00.872 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.872 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.872 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.872 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.872 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.873 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:04:00.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.873 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:00.873 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.873 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.873 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.873 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.873 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:04:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:04:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:04:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:04:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.874 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:04:00.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:04:00.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:04:00.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:04:00.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.875 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.875 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.875 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:04:00.875 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:04:00.875 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:04:00.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:04:00.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:04:00.876 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:04:00.876 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.876 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.876 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.876 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:04:00.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:04:00.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.877 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.877 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:04:00.877 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:04:00.877 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.877 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.877 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.877 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.877 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.877 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.877 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:00.877 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 287.48616888800001, 2024-08-30T16:04:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:04:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:04:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:04:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.879 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:04:00.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:04:00.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:04:00.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:04:00.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.880 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.880 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:04:00.880 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:04:00.880 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.880 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:04:00.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:04:00.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:04:00.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:04:00.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.881 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.881 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.881 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:04:00.881 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:04:00.881 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:04:00.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:04:00.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.882 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.882 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:04:00.882 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:04:00.882 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.882 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.882 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.882 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.882 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.883 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.883 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:00.883 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 287.48541930300001, 2024-08-30T16:04:00.883 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:04:00.883 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.883 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.883 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.883 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.883 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.883 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:04:00.883 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.884 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:00.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:00.884 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.884 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.884 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.884 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.884 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:04:00.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:00.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:04:00.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:04:00.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.885 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.885 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:04:00.885 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:04:00.885 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.885 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:04:00.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:04:00.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:04:00.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:04:00.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.886 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.886 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.886 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:04:00.886 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:04:00.886 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:04:00.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:04:00.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:04:00.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:04:00.887 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.887 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.887 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.887 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:04:00.887 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:04:00.888 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.888 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.888 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.888 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.888 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.888 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.888 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:00.889 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 287.45168777100002, 2024-08-30T16:04:00.889 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:04:00.889 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.890 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.890 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.890 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.890 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.890 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:04:00.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.890 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.890 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.890 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.890 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:00.890 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.890 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.891 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.891 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.891 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:00.891 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.891 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.892 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.892 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.892 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:04:00.892 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:04:00.892 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.893 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.893 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.893 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:04:00.893 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:04:00.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:04:00.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:04:00.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.894 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.894 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:04:00.894 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:04:00.894 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.894 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:04:00.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:04:00.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.895 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:04:00.895 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:04:00.895 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.895 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.895 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:04:00.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:04:00.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:04:00.896 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:04:00.896 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.896 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.896 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.896 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:04:00.897 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:04:00.897 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.897 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.897 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.897 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.897 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.898 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:00.898 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 287.45116176300002, 2024-08-30T16:04:00.898 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:04:00.898 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.899 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.899 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.899 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.899 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.899 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:04:00.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.900 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.900 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.900 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:00.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:00.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.901 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.901 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:04:00.901 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:00.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:04:00.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:04:00.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:04:00.902 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:04:00.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.902 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.902 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:04:00.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:04:00.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:04:00.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:04:00.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.903 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.903 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:04:00.903 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:04:00.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:04:00.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:04:00.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:04:00.904 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:04:00.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.904 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.904 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:04:00.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:04:00.905 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.905 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.905 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.905 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:00.905 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:00.905 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 287.45070620000001, 2024-08-30T16:04:00.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:04:00.905 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.905 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.905 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.906 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.906 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.906 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:04:00.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.906 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:00.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:00.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:04:00.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:04:00.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:04:00.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:04:00.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:04:00.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:04:00.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:04:00.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:04:00.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:04:00.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:04:00.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:04:00.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:04:00.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:04:00.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:04:00.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:04:00.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:04:00.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:04:00.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:04:00.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.911 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.912 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:04:00.912 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:00.912 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 282.78019088399998, 2024-08-30T16:04:00.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:04:00.912 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:00.912 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:00.912 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:00.912 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:00.912 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:00.913 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:04:00.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.913 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:00.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:00.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:00.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:00.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:00.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:00.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:00.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:04:00.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:00.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:00.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:04:00.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:04:00.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:00.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:04:00.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:04:00.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:00.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:04:00.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:04:00.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:00.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:04:00.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:04:00.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:00.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:04:00.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:04:00.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:00.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:04:00.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:04:00.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:00.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:04:00.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:04:00.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:00.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:00.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:00.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:04:00.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:04:00.917 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.918 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.918 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.918 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:00.919 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:00.919 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:00.920 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:00.921 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:00.918+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:01.021 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:01.020+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:01.085 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:01.086 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:01.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:01.122 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:01.121+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:01.217 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:01.217 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:01.223 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:01.221+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:01.227 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:01.324 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:01.322+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:01.392 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:01.392 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:01.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:01.424 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:01.422+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:01.525 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:01.524+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:01.538 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:01.538 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:01.538 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:01.538 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:01.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:01.626 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:01.625+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:01.675 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:01.675 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:01.675 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:01.675 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:01.675 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:01.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:01.727 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:01.725+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:01.828 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:01.827+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:01.855 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:01.855 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:01.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:01.855 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:04:01.855 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:01.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:01.929 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:01.928+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:01.993 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:01.993 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:01.994 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:01.994 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:02.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:02.045 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:02.044+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:02.131 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:02.130+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:02.172 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:02.173 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:02.173 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:02.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:02.267 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:02.265+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:02.332 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:02.331+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:02.355 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:02.356 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:02.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:02.356 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:02.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.356 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:02.356 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:02.356 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 281.08149750400003, 2024-08-30T16:04:02.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:04:02.356 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:02.356 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:02.357 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:02.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:02.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:02.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:02.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:02.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:02.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:02.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:02.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:04:02.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:04:02.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:02.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:04:02.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:04:02.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:02.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:04:02.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:04:02.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:02.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:04:02.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:04:02.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:02.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:04:02.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:04:02.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:02.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:04:02.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:04:02.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:02.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:04:02.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:04:02.360 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:02.361 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:02.361 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:02.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.361 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:02.361 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:02.361 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 280.808570143, 2024-08-30T16:04:02.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:04:02.361 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:02.361 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:02.361 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:02.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:02.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:02.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:02.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:02.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:02.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:02.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:02.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:04:02.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:04:02.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:02.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:04:02.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:04:02.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:02.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:04:02.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:04:02.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:02.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:04:02.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:04:02.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:02.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:04:02.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:04:02.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:02.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:04:02.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:04:02.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:02.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:04:02.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:04:02.365 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:02.365 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:02.365 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:02.365 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:02.365 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:02.365 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:02.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:02.451 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:02.450+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:02.534 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:02.533+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:02.539 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:02.539 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:02.539 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:02.539 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:02.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:02.635 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:02.634+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:02.715 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:02.715 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:02.715 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:02.715 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:02.716 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:02.724 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:02.767 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:02.764+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:02.837 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:02.835+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:02.894 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:02.894 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:02.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:02.894 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:02.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.894 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:04:02.895 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:02.895 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 277.51684962600001, 2024-08-30T16:04:02.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:04:02.895 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:02.895 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:02.895 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:02.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:02.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:02.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:02.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:02.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:02.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:02.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:02.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:04:02.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:04:02.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:02.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:04:02.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:04:02.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:02.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:04:02.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:04:02.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:02.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:04:02.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:04:02.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:02.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:04:02.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:04:02.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:02.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:04:02.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:04:02.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:02.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:04:02.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:04:02.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:02.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:02.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:02.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:04:02.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:04:02.899 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:02.899 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:02.900 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:02.900 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:02.900 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:02.900 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:02.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:02.947 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:02.946+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:03.033 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:03.033 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:03.033 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:03.033 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:03.039 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:03.037+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:03.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:03.118 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-08-30T16:04:03.140 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:03.138+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:03.209 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:03.210 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:03.210 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:03.210 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:03.210 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:03.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:03.241 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:03.239+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:03.342 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:03.341+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:03.412 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:03.413 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:03.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:03.413 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:03.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.414 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:03.414 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:03.414 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 278.95941354399997, 2024-08-30T16:04:03.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:04:03.414 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:03.414 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:03.415 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:03.415 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:03.415 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:03.415 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:03.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.415 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:03.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:03.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:03.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:03.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:03.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:03.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:04:03.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:04:03.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:03.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:04:03.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:04:03.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:03.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:04:03.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:04:03.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:03.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:04:03.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:04:03.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:03.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:04:03.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:04:03.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:04:03.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:04:03.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:04:03.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:03.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:04:03.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:04:03.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:03.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:04:03.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:04:03.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:04:03.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:04:03.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:04:03.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:03.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:04:03.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:04:03.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:03.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:04:03.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:04:03.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:03.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:04:03.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:04:03.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:03.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:04:03.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:04:03.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.428 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:03.428 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.428 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:04:03.428 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:03.428 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 277.84354720900001, 2024-08-30T16:04:03.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:04:03.428 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:03.428 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:03.428 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:03.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:04:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:03.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:03.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:04:03.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:04:03.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:03.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:04:03.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:04:03.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:03.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:04:03.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:04:03.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:03.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:04:03.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:04:03.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:04:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:04:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:04:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:04:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 277.78395571099998, 2024-08-30T16:04:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:04:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:04:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:03.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:04:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:03.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:04:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:04:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:04:03.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:04:03.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:03.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:04:03.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:04:03.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:03.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:04:03.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:04:03.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:04:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:04:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:04:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 277.75725702400001, 2024-08-30T16:04:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:04:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:03.437 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:03.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:04:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:04:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:04:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:04:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:04:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:04:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:04:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:04:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:04:03.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:04:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:04:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 276.88964049600003, 2024-08-30T16:04:03.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:04:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:04:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:04:03.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:04:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:04:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:04:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:04:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:04:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:04:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:04:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:04:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:04:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:04:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:04:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:04:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:04:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:04:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:04:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:04:03.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:04:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:04:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:04:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:04:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 276.73039341700002, 2024-08-30T16:04:03.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:04:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:04:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:04:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:04:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:04:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:04:03.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:04:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:04:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:04:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:04:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:04:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:03.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:04:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:04:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:04:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:04:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:03.451 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 276.45421935100001, 2024-08-30T16:04:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:04:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:03.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:04:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:04:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:04:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:04:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:04:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:04:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:04:03.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:04:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:04:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:04:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:04:03.455 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 275.79592777200003, 2024-08-30T16:04:03.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:04:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:04:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.457 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:03.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:03.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:03.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:03.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:03.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:03.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:04:03.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:03.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:04:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:04:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:04:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:04:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:04:03.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:04:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:04:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:04:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:04:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:04:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:04:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:04:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:04:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:04:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:04:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:04:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:04:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:04:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:04:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:04:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:04:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 275.70001448800002, 2024-08-30T16:04:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:04:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:03.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:03.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:03.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:04:03.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:03.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:03.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:04:03.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:04:03.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:03.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:04:03.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:04:03.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:03.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:04:03.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:04:03.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:03.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:04:03.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:04:03.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:03.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:04:03.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:04:03.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:03.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:04:03.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:04:03.467 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.467 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:03.467 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.468 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:03.468 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:03.468 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 216.44865397699999, 2024-08-30T16:04:03.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:04:03.468 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:03.468 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:03.468 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:03.468 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:03.468 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:03.468 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:03.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:03.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:04:03.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:04:03.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:03.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:04:03.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:04:03.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:03.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:04:03.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:04:03.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:03.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:04:03.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:04:03.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:03.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:04:03.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:04:03.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:03.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:04:03.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:04:03.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.472 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:03.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.472 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:04:03.472 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:03.472 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 156.43954929, 2024-08-30T16:04:03.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:04:03.472 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:03.472 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:03.472 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:03.472 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:03.473 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:03.473 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:03.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.473 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:03.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:03.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:03.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:03.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:03.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:03.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:03.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:04:03.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:04:03.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:03.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:04:03.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:04:03.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:03.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:04:03.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:04:03.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:03.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:04:03.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:04:03.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:03.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:04:03.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:04:03.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:03.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:03.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:03.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:04:03.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:04:03.476 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.476 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:03.476 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.476 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:03.476 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:03.476 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:03.479 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:03.480 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:03.477+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:03.580 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:03.578+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:03.618 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:03.618 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:03.618 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:03.628 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:03.680 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:03.678+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:03.752 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:03.752 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:03.752 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:03.753 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:03.753 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:03.762 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:03.781 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:03.779+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:03.882 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:03.880+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:03.925 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:03.925 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:03.926 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:04:03.926 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:03.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:03.982 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:03.980+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:04.063 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:04.064 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:04.064 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:04.064 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:04.073 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:04.115 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:04.113+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:04.185 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:04.183+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:04.239 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:04.239 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:04.240 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:04.240 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:04.250 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:04.284 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:04.283+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:04.385 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:04.383+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:04.459 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:04.460 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:04.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:04.460 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:04.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.460 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:04.460 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:04.460 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 293.59858855599998, 2024-08-30T16:04:04.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:04:04.460 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.460 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.460 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.460 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.461 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.461 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:04:04.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.461 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:04.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:04.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:04:04.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:04:04.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:04:04.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:04:04.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:04:04.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:04:04.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:04:04.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:04:04.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.463 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.463 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:04:04.463 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:04:04.463 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.463 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:04:04.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:04:04.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:04:04.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:04:04.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:04:04.464 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.464 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.464 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.464 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:04:04.464 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:04:04.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:04:04.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:04:04.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.465 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:04:04.465 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:04:04.465 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.465 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.465 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.466 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:04:04.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:04:04.466 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.466 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.466 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.466 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.466 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.466 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:04.466 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:04.466 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 292.682172536, 2024-08-30T16:04:04.466 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:04:04.466 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.466 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.467 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.467 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:04:04.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.467 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:04.467 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.467 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.467 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.467 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.467 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:04.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:04:04.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:04.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.468 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.468 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:04:04.468 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:04:04.468 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.468 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:04:04.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:04:04.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:04:04.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:04:04.469 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.469 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.469 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.469 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:04:04.469 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:04:04.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:04:04.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:04:04.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:04:04.470 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:04:04.470 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.470 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.470 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.470 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:04:04.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:04:04.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.471 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.471 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:04:04.471 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:04:04.471 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.471 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.471 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.471 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.471 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.471 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:04.471 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:04.472 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 292.68046798300003, 2024-08-30T16:04:04.472 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:04:04.472 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.472 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.472 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.472 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.472 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.472 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:04:04.472 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.472 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.472 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.472 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.472 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:04.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:04.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.473 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:04:04.473 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:04:04.473 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.473 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.473 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:04:04.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:04:04.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:04:04.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:04:04.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.474 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.474 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.474 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:04:04.474 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:04:04.474 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:04:04.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:04:04.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:04:04.475 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:04:04.475 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.475 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.475 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.475 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:04:04.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:04:04.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:04:04.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:04:04.476 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.476 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.476 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.476 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:04:04.476 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:04:04.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.476 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.476 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.477 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.477 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.477 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:04.477 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 292.261014176, 2024-08-30T16:04:04.477 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:04:04.477 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.477 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.477 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.477 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.477 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.477 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:04:04.477 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.478 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:04.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.478 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:04.478 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.478 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.478 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.478 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:04:04.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:04:04.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:04:04.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:04:04.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.479 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.479 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:04:04.479 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:04:04.479 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.479 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:04:04.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:04:04.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:04:04.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:04:04.480 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.480 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.480 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.480 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:04:04.480 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:04:04.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:04:04.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:04:04.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.481 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:04:04.481 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:04:04.481 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.481 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.481 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.482 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:04:04.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:04:04.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.482 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.482 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.482 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.482 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.482 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.482 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:04.482 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 292.24412600199997, 2024-08-30T16:04:04.482 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:04:04.482 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.482 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.483 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.483 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.483 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.483 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:04:04.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.483 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.483 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:04.483 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.483 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.483 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.483 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:04.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:04:04.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:04:04.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.484 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.484 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.484 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:04:04.484 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:04:04.484 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:04:04.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:04:04.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:04:04.485 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:04:04.485 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.485 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.485 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.485 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:04:04.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:04:04.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:04:04.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:04:04.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.486 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.486 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:04:04.486 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:04:04.486 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.486 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:04:04.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:04:04.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.487 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.487 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:04:04.487 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:04:04.487 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.487 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.487 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.487 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.487 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.487 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.488 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:04.488 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 292.23112593299999, 2024-08-30T16:04:04.488 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:04:04.488 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.488 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.488 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.488 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.488 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.488 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:04:04.488 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.488 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.488 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.488 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:04.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:04.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.489 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.489 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.489 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:04:04.489 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:04:04.489 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:04:04.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:04:04.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:04:04.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:04:04.490 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.490 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.490 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.490 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:04:04.490 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:04:04.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:04:04.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:04:04.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.491 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:04:04.491 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:04:04.491 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.491 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.491 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:04:04.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:04:04.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:04:04.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:04:04.492 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.492 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.492 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.492 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:04:04.492 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:04:04.492 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.493 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.493 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.493 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.493 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.493 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.493 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:04.493 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 292.22863364099999, 2024-08-30T16:04:04.493 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:04:04.493 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.493 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.493 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.493 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.493 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.494 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:04:04.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.494 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:04.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.494 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.494 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:04.494 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.494 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.494 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:04:04.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:04.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:04:04.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:04:04.495 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.495 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.495 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.495 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:04:04.495 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:04:04.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:04:04.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:04:04.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.496 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:04:04.496 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:04:04.496 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.496 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.496 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:04:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:04:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:04:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:04:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:04:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:04:04.497 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.498 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.498 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:04:04.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:04:04.498 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.498 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.498 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.498 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.498 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.498 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.498 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:04.498 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 292.22787619299999, 2024-08-30T16:04:04.498 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:04:04.498 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.499 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.499 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.499 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.499 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.499 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:04:04.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.499 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.499 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.499 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:04.499 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.499 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.499 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:04.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:04:04.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:04:04.500 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.500 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.500 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.500 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:04:04.500 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:04:04.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:04:04.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:04:04.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:04:04.501 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:04:04.501 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.501 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.501 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.501 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:04:04.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:04:04.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:04:04.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:04:04.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.502 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.502 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:04:04.502 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:04:04.502 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.502 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:04:04.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:04:04.503 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.503 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.503 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.503 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:04:04.503 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:04:04.503 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.503 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.504 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.504 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.504 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.504 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.504 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:04.504 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 292.227143743, 2024-08-30T16:04:04.504 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:04:04.504 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.504 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.504 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.504 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:04:04.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.505 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:04.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.505 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.505 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.505 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.505 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:04.505 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:04:04.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:04.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.506 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.506 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:04:04.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:04:04.506 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.506 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:04:04.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:04:04.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.507 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:04:04.507 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:04:04.507 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.507 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.507 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:04:04.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:04:04.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:04:04.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:04:04.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.508 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.508 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.508 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:04:04.508 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:04:04.508 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:04:04.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:04:04.509 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.509 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.509 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.509 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:04:04.509 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:04:04.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.509 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.509 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.510 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.510 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.510 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:04.510 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 292.22460494400002, 2024-08-30T16:04:04.510 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:04:04.510 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.510 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.510 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.510 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:04:04.510 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.511 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:04.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.511 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:04.511 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.511 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.511 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:04:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:04:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:04:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:04:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:04:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:04:04.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:04:04.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:04:04.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:04:04.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.513 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.513 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:04:04.513 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:04:04.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:04:04.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:04:04.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.514 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:04:04.514 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:04:04.514 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.514 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.514 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.515 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:04:04.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:04:04.515 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.515 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.515 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.515 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.515 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.515 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.515 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:04.515 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 292.22385694299999, 2024-08-30T16:04:04.515 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:04:04.515 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.515 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.516 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.516 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.516 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.516 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:04:04.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.516 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.516 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:04.516 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.516 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.516 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.516 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:04.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:04:04.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:04:04.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.517 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.517 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:04:04.517 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:04:04.517 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.517 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:04:04.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:04:04.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.518 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.518 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.518 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:04:04.518 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:04:04.518 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:04:04.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:04:04.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:04:04.519 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:04:04.519 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.519 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.519 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.519 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:04:04.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:04:04.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:04:04.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:04:04.520 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.520 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.520 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.520 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:04:04.520 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:04:04.520 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.520 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.521 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.521 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.521 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.521 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.521 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:04.521 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 291.45174047099999, 2024-08-30T16:04:04.521 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:04:04.521 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.521 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.521 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.521 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.521 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.521 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:04:04.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.522 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:04.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.522 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:04.522 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.522 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.522 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.522 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:04:04.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:04.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:04:04.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:04:04.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.523 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.523 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.523 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:04:04.523 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:04:04.523 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:04:04.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:04:04.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:04:04.524 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:04:04.524 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.524 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.524 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.524 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:04:04.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:04:04.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:04:04.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:04:04.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.525 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.525 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:04:04.525 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:04:04.525 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.525 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.526 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.526 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:04:04.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:04:04.526 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.526 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.526 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.526 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.526 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.526 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.526 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:04.526 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 291.44968332500002, 2024-08-30T16:04:04.526 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:04:04.526 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.526 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.527 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.527 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:04:04.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.527 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.527 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:04.527 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.527 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.527 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.527 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:04.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:04:04.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:04.528 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.528 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.528 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.528 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:04:04.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:04:04.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:04:04.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:04:04.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.529 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:04:04.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:04:04.529 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.529 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:04:04.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:04:04.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:04:04.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:04:04.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:04:04.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:04:04.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:04:04.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:04:04.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:04:04.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:04:04.531 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.531 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.531 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.532 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.532 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:04.532 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 291.18673882500002, 2024-08-30T16:04:04.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:04:04.532 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.532 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.532 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.532 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.532 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.532 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:04:04.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.532 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:04.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:04.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:04:04.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:04:04.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:04:04.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:04:04.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:04:04.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:04:04.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:04:04.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:04:04.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:04:04.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:04:04.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:04:04.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:04:04.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:04:04.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:04:04.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:04:04.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:04:04.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.536 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.536 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:04:04.536 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:04:04.537 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.537 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.537 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.537 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.537 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:04.537 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 291.18564732200002, 2024-08-30T16:04:04.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:04:04.537 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.537 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.537 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:04:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:04.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:04:04.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:04.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:04:04.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:04:04.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:04:04.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:04:04.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:04:04.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:04:04.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:04:04.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:04:04.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:04:04.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:04:04.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:04:04.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:04:04.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.541 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.541 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:04:04.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:04:04.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:04:04.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:04:04.542 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.542 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.542 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.542 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.542 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:04.542 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 291.18489773700003, 2024-08-30T16:04:04.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:04:04.543 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.543 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.543 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.543 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:04:04.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.543 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:04.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:04.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:04:04.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:04.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:04:04.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:04:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:04:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:04:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:04:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:04:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:04:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:04:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:04:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:04:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:04:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:04:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:04:04.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:04:04.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.547 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.547 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:04:04.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:04:04.547 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.547 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.547 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.547 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.548 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:04.548 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 291.15116620499998, 2024-08-30T16:04:04.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:04:04.548 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.548 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.548 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.548 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.548 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.548 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:04:04.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.548 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:04.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:04.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:04:04.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:04:04.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:04:04.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:04:04.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:04:04.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:04:04.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:04:04.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:04:04.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:04:04.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:04:04.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:04:04.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:04:04.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:04:04.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:04:04.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.552 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:04:04.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:04:04.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.552 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:04:04.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:04:04.553 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.553 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.553 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.553 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.553 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:04.553 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 291.15064019699997, 2024-08-30T16:04:04.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:04:04.554 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.554 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.554 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.554 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.554 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.554 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:04:04.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.554 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:04.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:04.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:04:04.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:04.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:04:04.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:04:04.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:04:04.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:04:04.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:04:04.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:04:04.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:04:04.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:04:04.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:04:04.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:04:04.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:04:04.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:04:04.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:04:04.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:04:04.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.558 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.558 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:04:04.558 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:04:04.558 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.558 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.558 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.559 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:04.559 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:04.559 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 291.15018463400003, 2024-08-30T16:04:04.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:04:04.559 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.559 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.559 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.559 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.559 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.559 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:04:04.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.559 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:04.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:04.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:04:04.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:04:04.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:04:04.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:04:04.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:04:04.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:04:04.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:04:04.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:04:04.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:04:04.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:04:04.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:04:04.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:04:04.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:04:04.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:04:04.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:04:04.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:04:04.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:04:04.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:04:04.564 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.564 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.564 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.564 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:04:04.564 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:04.564 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 286.47966931799999, 2024-08-30T16:04:04.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:04:04.564 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:04.564 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:04.564 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:04.564 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:04.565 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:04.565 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:04:04.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.565 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:04.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:04.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:04.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:04.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:04.565 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:04.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:04.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:04:04.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:04.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:04.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:04:04.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:04:04.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:04.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:04:04.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:04:04.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:04.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:04:04.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:04:04.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:04.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:04:04.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:04:04.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:04.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:04:04.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:04:04.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:04.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:04:04.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:04:04.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:04.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:04:04.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:04:04.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:04.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:04.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:04.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:04:04.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:04:04.569 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.569 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.569 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.569 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:04.569 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:04.569 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:04.571 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:04.571 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:04.569+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:04.684 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:04.682+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:04.697 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:04.697 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:04.706 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:04.773 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:04.771+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:04.871 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:04.872 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:04.873 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:04.872+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:04.881 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:04.975 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:04.973+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:05.003 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:05.003 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:05.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:05.076 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:05.074+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:05.155 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:05.156 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:05.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:05.210 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:05.208+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:05.281 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:05.277+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:05.295 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:05.296 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:05.296 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:05.296 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:05.296 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:05.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:05.390 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:05.388+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:05.473 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:05.473 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2024-08-30T16:04:05.473 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:05.480 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:05.479+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:05.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:05.581 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:05.580+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:05.652 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:05.652 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:05.652 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:05.652 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:05.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:05.702 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:05.701+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:05.783 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:05.781+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:05.829 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:05.829 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:05.829 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:05.829 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:05.830 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:05.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:05.884 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:05.883+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:05.970 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:05.971 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:05.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:05.971 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:05.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.971 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:05.971 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:05.971 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 284.69612961500002, 2024-08-30T16:04:05.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:04:05.971 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:05.971 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:05.971 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:05.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:05.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:05.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:05.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:05.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:05.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:05.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:05.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:04:05.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:04:05.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:05.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:04:05.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:04:05.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:05.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:04:05.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:04:05.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:05.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:04:05.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:04:05.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:05.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:04:05.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:04:05.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:05.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:04:05.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:04:05.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:05.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:04:05.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:04:05.975 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:05.975 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:05.975 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:05.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.975 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:05.975 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:05.975 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 284.42320225399999, 2024-08-30T16:04:05.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:04:05.975 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:05.976 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:05.976 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:05.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:05.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:05.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:05.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:05.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:05.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:05.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:05.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:04:05.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:04:05.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:05.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:04:05.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:04:05.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:05.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:04:05.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:04:05.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:05.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:04:05.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:04:05.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:05.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:04:05.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:04:05.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:05.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:04:05.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:04:05.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:05.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:05.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:05.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:04:05.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:04:05.979 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:05.979 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:05.979 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:05.979 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:05.979 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:05.979 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:05.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:06.022 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:06.020+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:06.088 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:06.086+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:06.148 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:06.279 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:06.279 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:06.279 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:06.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:06.282 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:06.279+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:06.395 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:06.394+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:06.409 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:06.409 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:06.409 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:06.409 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:06.409 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:06.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:06.483 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:06.481+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:06.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:06.589 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:06.588+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:06.607 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:06.607 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:06.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:06.607 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:06.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:06.608 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:04:06.608 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:06.608 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 281.228297661, 2024-08-30T16:04:06.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:04:06.608 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:06.608 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:06.608 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:06.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:06.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:06.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:06.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:06.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:06.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:06.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:06.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:06.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:06.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:06.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:06.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:06.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:04:06.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:04:06.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:06.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:06.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:06.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:04:06.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:04:06.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:06.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:06.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:06.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:04:06.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:04:06.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:06.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:06.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:06.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:04:06.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:04:06.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:04:06.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:06.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:06.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:06.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:04:06.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:04:06.612 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:06.612 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:06.612 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:06.612 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:06.612 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:06.612 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:06.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:06.629 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:06.629 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:06.633 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:06.633 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:06.685 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:06.683+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:06.772 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:06.773 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:06.773 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:06.773 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:06.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:06.786 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:06.784+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:06.886 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:06.885+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:06.914 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:06.914 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:06.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:07.007 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:07.006+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:07.088 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:07.087+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:07.123 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 282.67014441800001, 2024-08-30T16:04:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:04:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:07.123 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:04:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:04:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:04:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:04:07.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:04:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:04:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:04:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:04:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:04:07.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:04:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:04:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:04:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:04:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:04:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:04:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:07.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:04:07.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:04:07.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:04:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:04:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:04:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:04:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:04:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:04:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:04:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:04:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:04:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:04:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:04:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.130 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:04:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 281.55427808299999, 2024-08-30T16:04:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:04:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:04:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:04:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:04:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:04:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:04:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:04:07.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:04:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:04:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:04:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:04:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:04:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:04:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:04:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 281.49468658500001, 2024-08-30T16:04:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:04:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:07.135 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:04:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:07.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:04:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:04:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:04:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:04:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:04:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:04:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:04:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:04:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:04:07.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:04:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:04:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:04:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:07.139 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 281.46798789799999, 2024-08-30T16:04:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:04:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:07.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:04:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:07.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:04:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:04:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:04:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:04:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:04:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:04:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:04:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:04:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:04:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:04:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.143 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 280.60037137, 2024-08-30T16:04:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:04:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:04:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:04:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:04:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:04:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:04:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:04:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:04:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:07.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:04:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:04:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:04:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:04:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:04:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:04:07.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:04:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:04:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:04:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:04:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:04:07.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:04:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:04:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:04:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:04:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:07.150 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 280.44112429099999, 2024-08-30T16:04:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:04:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:04:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:04:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:04:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:04:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:04:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:04:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:04:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:04:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:04:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:04:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:04:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:04:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:04:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:04:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 280.16495022499998, 2024-08-30T16:04:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:04:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:04:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:04:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:04:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:04:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:04:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:04:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:04:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:04:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:04:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:04:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:04:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 279.50665864600001, 2024-08-30T16:04:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:04:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:04:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:04:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:04:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:04:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:04:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:04:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:04:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:04:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:04:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:04:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:04:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:04:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:04:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:04:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:04:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:04:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:04:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:04:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:04:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:04:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:04:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:04:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:04:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 279.410745362, 2024-08-30T16:04:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:04:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:07.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:04:07.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:07.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:07.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:04:07.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:04:07.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:07.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:04:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:04:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:04:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:04:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:04:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:04:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:07.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:04:07.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:04:07.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:07.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:04:07.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:04:07.169 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.169 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.169 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.169 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:07.170 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:07.170 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 220.159384851, 2024-08-30T16:04:07.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:04:07.170 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:07.170 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:07.170 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:07.170 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:07.170 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:07.170 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:07.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.170 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:07.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:04:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:04:07.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:07.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:04:07.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:04:07.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:07.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:04:07.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:04:07.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:07.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:04:07.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:04:07.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:07.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:04:07.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:04:07.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:07.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:04:07.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:04:07.173 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.173 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.174 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.174 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:04:07.174 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:07.174 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 160.15028016400001, 2024-08-30T16:04:07.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:04:07.174 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:07.174 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:07.174 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:07.174 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:07.174 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:07.174 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:07.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.175 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:07.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:07.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:07.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:07.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:07.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:07.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:07.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:04:07.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:04:07.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:07.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:04:07.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:04:07.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:07.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:04:07.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:04:07.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:07.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:04:07.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:04:07.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:07.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:04:07.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:04:07.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:07.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:04:07.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:04:07.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.178 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.178 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.178 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:07.179 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:07.224 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:07.222+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:07.290 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:07.288+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:07.305 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:07.305 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:07.306 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:07.306 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:07.318 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:07.388 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:04:07.388 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:07.388 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 82, 2024-08-30T16:04:07.388 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "9d72e2e2-0554-42af-8e62-9b1a3a804112", 2024-08-30T16:04:07.388 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-08-30T15:58:20.129410+0000", 2024-08-30T16:04:07.389 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-08-30T16:03:50.972329+0000", 2024-08-30T16:04:07.389 INFO:teuthology.orchestra.run.smithi050.stdout: "last_up_change": "2024-08-30T16:03:35.238525+0000", 2024-08-30T16:04:07.389 INFO:teuthology.orchestra.run.smithi050.stdout: "last_in_change": "2024-08-30T16:03:05.847464+0000", 2024-08-30T16:04:07.389 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T16:04:07.389 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_num": 5799936, 2024-08-30T16:04:07.389 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_set": [ 2024-08-30T16:04:07.389 INFO:teuthology.orchestra.run.smithi050.stdout: "pglog_hardlimit", 2024-08-30T16:04:07.389 INFO:teuthology.orchestra.run.smithi050.stdout: "purged_snapdirs", 2024-08-30T16:04:07.389 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_deletes", 2024-08-30T16:04:07.389 INFO:teuthology.orchestra.run.smithi050.stdout: "sortbitwise" 2024-08-30T16:04:07.389 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:04:07.389 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_version": 7, 2024-08-30T16:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T16:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T16:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T16:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_snapshot": "", 2024-08-30T16:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_max": 2, 2024-08-30T16:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd": 8, 2024-08-30T16:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout: "require_min_compat_client": "luminous", 2024-08-30T16:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout: "min_compat_client": "luminous", 2024-08-30T16:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout: "require_osd_release": "reef", 2024-08-30T16:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_crimson": false, 2024-08-30T16:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout: "pools": [ 2024-08-30T16:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 1, 2024-08-30T16:04:07.391 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".mgr", 2024-08-30T16:04:07.391 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-08-30T15:59:08.355540+0000", 2024-08-30T16:04:07.391 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-08-30T16:04:07.391 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-08-30T16:04:07.391 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-08-30T16:04:07.391 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 2, 2024-08-30T16:04:07.391 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 1, 2024-08-30T16:04:07.391 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-08-30T16:04:07.391 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-08-30T16:04:07.391 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-08-30T16:04:07.391 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T16:04:07.391 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T16:04:07.391 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-08-30T16:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-08-30T16:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 1, 2024-08-30T16:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 1, 2024-08-30T16:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 1, 2024-08-30T16:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 1, 2024-08-30T16:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 1, 2024-08-30T16:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-08-30T16:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-08-30T16:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-08-30T16:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-08-30T16:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-08-30T16:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-08-30T16:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-08-30T16:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "19", 2024-08-30T16:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-08-30T16:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T16:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T16:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-08-30T16:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-08-30T16:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-08-30T16:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-08-30T16:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-08-30T16:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-08-30T16:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-08-30T16:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-08-30T16:04:07.394 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-08-30T16:04:07.394 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-08-30T16:04:07.394 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-08-30T16:04:07.394 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-08-30T16:04:07.394 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-08-30T16:04:07.394 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-08-30T16:04:07.394 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-08-30T16:04:07.394 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T16:04:07.394 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T16:04:07.394 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T16:04:07.394 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-08-30T16:04:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-08-30T16:04:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-08-30T16:04:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-08-30T16:04:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-08-30T16:04:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-08-30T16:04:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-08-30T16:04:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-08-30T16:04:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-08-30T16:04:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-08-30T16:04:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T16:04:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-08-30T16:04:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-08-30T16:04:07.395 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-08-30T16:04:07.396 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-08-30T16:04:07.396 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-08-30T16:04:07.396 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-08-30T16:04:07.396 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_max": 32, 2024-08-30T16:04:07.396 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_min": 1 2024-08-30T16:04:07.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.396 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-08-30T16:04:07.396 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": {} 2024-08-30T16:04:07.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.396 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-08-30T16:04:07.396 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 8, 2024-08-30T16:04:07.396 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 8, 2024-08-30T16:04:07.397 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.25, 2024-08-30T16:04:07.397 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-08-30T16:04:07.397 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-08-30T16:04:07.397 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-08-30T16:04:07.397 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-08-30T16:04:07.397 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-08-30T16:04:07.397 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.397 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-08-30T16:04:07.397 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "rbd", 2024-08-30T16:04:07.397 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-08-30T15:59:21.696796+0000", 2024-08-30T16:04:07.397 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-08-30T16:04:07.398 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-08-30T16:04:07.398 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-08-30T16:04:07.398 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 2, 2024-08-30T16:04:07.398 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 1, 2024-08-30T16:04:07.398 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-08-30T16:04:07.398 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-08-30T16:04:07.398 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-08-30T16:04:07.398 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T16:04:07.398 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T16:04:07.398 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-08-30T16:04:07.398 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-08-30T16:04:07.398 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 8, 2024-08-30T16:04:07.398 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 8, 2024-08-30T16:04:07.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 8, 2024-08-30T16:04:07.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 8, 2024-08-30T16:04:07.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 8, 2024-08-30T16:04:07.399 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-08-30T16:04:07.399 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-08-30T16:04:07.399 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-08-30T16:04:07.399 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-08-30T16:04:07.399 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-08-30T16:04:07.399 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-08-30T16:04:07.400 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-08-30T16:04:07.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "28", 2024-08-30T16:04:07.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-08-30T16:04:07.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T16:04:07.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T16:04:07.400 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-08-30T16:04:07.400 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-08-30T16:04:07.400 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 3, 2024-08-30T16:04:07.400 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 28, 2024-08-30T16:04:07.400 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-08-30T16:04:07.400 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-08-30T16:04:07.400 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-08-30T16:04:07.401 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-08-30T16:04:07.401 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-08-30T16:04:07.401 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-08-30T16:04:07.401 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-08-30T16:04:07.401 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-08-30T16:04:07.401 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-08-30T16:04:07.401 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-08-30T16:04:07.401 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-08-30T16:04:07.401 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T16:04:07.401 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T16:04:07.401 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T16:04:07.401 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-08-30T16:04:07.402 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-08-30T16:04:07.402 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-08-30T16:04:07.402 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-08-30T16:04:07.402 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-08-30T16:04:07.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.402 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-08-30T16:04:07.402 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-08-30T16:04:07.402 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-08-30T16:04:07.402 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-08-30T16:04:07.402 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-08-30T16:04:07.402 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T16:04:07.402 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-08-30T16:04:07.402 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-08-30T16:04:07.403 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-08-30T16:04:07.403 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-08-30T16:04:07.403 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-08-30T16:04:07.403 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-08-30T16:04:07.403 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-08-30T16:04:07.403 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd": {} 2024-08-30T16:04:07.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.403 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-08-30T16:04:07.403 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 3.9700000286102295, 2024-08-30T16:04:07.403 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 3.9700000286102295, 2024-08-30T16:04:07.404 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.62999999523162842, 2024-08-30T16:04:07.404 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2.5, 2024-08-30T16:04:07.404 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2.5, 2024-08-30T16:04:07.404 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-08-30T16:04:07.404 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-08-30T16:04:07.404 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-08-30T16:04:07.404 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.405 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.405 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:04:07.405 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-08-30T16:04:07.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.405 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-08-30T16:04:07.405 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4", 2024-08-30T16:04:07.405 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:04:07.405 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:04:07.405 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:04:07.405 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:07.405 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:07.405 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:07.406 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 15, 2024-08-30T16:04:07.406 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 60, 2024-08-30T16:04:07.406 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:04:07.406 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:07.406 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:07.406 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.406 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.406 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6812", 2024-08-30T16:04:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:04:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6813", 2024-08-30T16:04:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:04:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.408 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:07.408 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.408 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.408 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6815", 2024-08-30T16:04:07.408 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:04:07.408 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.408 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.408 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:07.408 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.408 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.409 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6814", 2024-08-30T16:04:07.409 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:04:07.409 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.409 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.409 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6812/1957399494", 2024-08-30T16:04:07.409 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6813/1957399494", 2024-08-30T16:04:07.409 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6815/1957399494", 2024-08-30T16:04:07.409 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6814/1957399494", 2024-08-30T16:04:07.409 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:07.409 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:04:07.410 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:04:07.410 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.410 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-08-30T16:04:07.410 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "0fe11a1c-4326-4bfd-a2a4-bff14023f301", 2024-08-30T16:04:07.410 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:04:07.410 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:04:07.410 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:04:07.410 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:07.410 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:07.410 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:07.410 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 22, 2024-08-30T16:04:07.411 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 22, 2024-08-30T16:04:07.411 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:04:07.411 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:07.411 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:07.411 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.411 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.411 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6808", 2024-08-30T16:04:07.411 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:04:07.411 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.411 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.411 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:07.412 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.412 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.412 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6809", 2024-08-30T16:04:07.412 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:04:07.412 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.412 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.412 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:07.412 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.412 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.412 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6811", 2024-08-30T16:04:07.413 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:04:07.413 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.413 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.413 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:07.413 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.413 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.413 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6810", 2024-08-30T16:04:07.413 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:04:07.413 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.413 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.414 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6808/3208178756", 2024-08-30T16:04:07.414 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6809/3208178756", 2024-08-30T16:04:07.414 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6811/3208178756", 2024-08-30T16:04:07.415 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6810/3208178756", 2024-08-30T16:04:07.415 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:07.415 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:04:07.415 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:04:07.415 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.415 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-08-30T16:04:07.415 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "a9d5354b-3600-41ed-80c5-a3439da7aca6", 2024-08-30T16:04:07.415 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:04:07.415 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:04:07.416 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:04:07.416 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:07.416 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:07.416 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:07.417 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 16, 2024-08-30T16:04:07.417 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:04:07.417 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:04:07.417 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:07.417 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:07.418 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.418 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.418 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6804", 2024-08-30T16:04:07.418 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:04:07.418 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.418 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.418 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:07.418 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.419 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.419 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6805", 2024-08-30T16:04:07.419 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:04:07.419 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.419 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.419 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:07.419 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.420 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.420 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6807", 2024-08-30T16:04:07.420 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:04:07.420 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.420 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.421 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:07.421 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.421 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.422 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6806", 2024-08-30T16:04:07.422 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:04:07.422 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.422 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.422 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6804/287018800", 2024-08-30T16:04:07.422 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6805/287018800", 2024-08-30T16:04:07.422 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6807/287018800", 2024-08-30T16:04:07.422 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6806/287018800", 2024-08-30T16:04:07.422 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:07.422 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:04:07.423 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:04:07.423 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.423 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-08-30T16:04:07.423 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "78bcc398-a7e4-4efc-9836-1dfb94407c9c", 2024-08-30T16:04:07.423 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:04:07.424 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:04:07.424 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:04:07.425 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:07.425 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:07.425 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:07.425 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:04:07.425 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 64, 2024-08-30T16:04:07.425 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:04:07.425 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:07.425 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:07.425 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.426 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.426 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6800", 2024-08-30T16:04:07.426 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:04:07.426 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.426 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.426 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:07.426 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.427 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.427 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6801", 2024-08-30T16:04:07.427 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:04:07.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.427 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.427 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:07.427 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.427 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.427 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6803", 2024-08-30T16:04:07.428 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:04:07.428 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.428 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.428 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:07.428 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.428 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.428 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6802", 2024-08-30T16:04:07.428 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:04:07.428 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.429 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.429 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6800/1140717714", 2024-08-30T16:04:07.429 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6801/1140717714", 2024-08-30T16:04:07.429 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6803/1140717714", 2024-08-30T16:04:07.429 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6802/1140717714", 2024-08-30T16:04:07.429 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:07.429 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:04:07.429 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:04:07.429 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.430 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-08-30T16:04:07.430 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "53a1060b-7985-4c2c-820e-64b158056113", 2024-08-30T16:04:07.430 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:04:07.430 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:04:07.430 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:04:07.430 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:07.430 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:07.430 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:07.430 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 20, 2024-08-30T16:04:07.430 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 24, 2024-08-30T16:04:07.431 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:04:07.431 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:07.431 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:07.431 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.431 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.431 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6805", 2024-08-30T16:04:07.431 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:04:07.431 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.431 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.432 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:07.432 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.432 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.432 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6806", 2024-08-30T16:04:07.432 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:04:07.432 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.432 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.432 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:07.432 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.433 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.433 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6808", 2024-08-30T16:04:07.433 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:04:07.433 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.433 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.433 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:07.433 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.433 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.433 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6807", 2024-08-30T16:04:07.433 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:04:07.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.434 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.434 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6805/1468073345", 2024-08-30T16:04:07.434 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6806/1468073345", 2024-08-30T16:04:07.434 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6808/1468073345", 2024-08-30T16:04:07.434 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6807/1468073345", 2024-08-30T16:04:07.434 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:07.434 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:04:07.434 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:04:07.434 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.435 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-08-30T16:04:07.435 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "853a03f7-227d-4f98-9e5c-c29896820746", 2024-08-30T16:04:07.435 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:04:07.435 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:04:07.435 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:04:07.435 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:07.435 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:07.435 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:07.435 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 13, 2024-08-30T16:04:07.435 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 64, 2024-08-30T16:04:07.435 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:04:07.436 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:07.436 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:07.436 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.436 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.436 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6809", 2024-08-30T16:04:07.436 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:04:07.436 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.436 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.436 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:07.436 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.437 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.437 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6810", 2024-08-30T16:04:07.437 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:04:07.437 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.437 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.437 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:07.437 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.438 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.438 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6812", 2024-08-30T16:04:07.438 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:04:07.438 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.438 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.438 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:07.438 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.439 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.439 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6811", 2024-08-30T16:04:07.439 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:04:07.439 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.439 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.440 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6809/3876025616", 2024-08-30T16:04:07.440 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6810/3876025616", 2024-08-30T16:04:07.440 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6812/3876025616", 2024-08-30T16:04:07.440 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6811/3876025616", 2024-08-30T16:04:07.440 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:07.440 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:04:07.440 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:04:07.440 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.440 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-08-30T16:04:07.440 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "bc978060-bf28-4027-8bba-7d6d8659bfc5", 2024-08-30T16:04:07.440 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 0, 2024-08-30T16:04:07.441 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:04:07.441 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:04:07.441 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:07.441 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:07.441 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:07.441 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:04:07.441 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:04:07.441 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 75, 2024-08-30T16:04:07.441 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:07.441 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:07.441 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.441 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.442 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6801", 2024-08-30T16:04:07.442 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:04:07.442 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.442 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.442 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:07.442 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.442 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.442 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6802", 2024-08-30T16:04:07.442 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:04:07.442 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.442 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.443 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:07.443 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.443 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.443 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6804", 2024-08-30T16:04:07.443 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:04:07.443 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.443 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.443 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:07.443 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.443 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.444 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6803", 2024-08-30T16:04:07.444 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:04:07.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.444 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.444 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6801/128519122", 2024-08-30T16:04:07.444 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6802/128519122", 2024-08-30T16:04:07.444 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6804/128519122", 2024-08-30T16:04:07.444 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6803/128519122", 2024-08-30T16:04:07.444 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:07.444 INFO:teuthology.orchestra.run.smithi050.stdout: "exists" 2024-08-30T16:04:07.444 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.445 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-08-30T16:04:07.445 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "45df5c6f-b124-46d1-9efb-62b78bc584ed", 2024-08-30T16:04:07.445 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:04:07.445 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:04:07.445 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:04:07.445 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:07.445 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:07.445 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:07.445 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:04:07.446 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 40, 2024-08-30T16:04:07.446 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:04:07.446 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:07.446 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:07.446 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.446 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.446 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6813", 2024-08-30T16:04:07.446 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:04:07.446 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.446 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.446 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:07.447 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.447 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.447 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6814", 2024-08-30T16:04:07.447 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:04:07.447 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.447 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.447 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:07.447 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.447 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.447 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6816", 2024-08-30T16:04:07.447 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:04:07.448 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.448 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.448 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:07.448 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:07.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.448 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:07.448 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6815", 2024-08-30T16:04:07.448 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:04:07.448 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.448 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.448 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6813/68721277", 2024-08-30T16:04:07.449 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6814/68721277", 2024-08-30T16:04:07.449 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6816/68721277", 2024-08-30T16:04:07.449 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6815/68721277", 2024-08-30T16:04:07.449 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:07.449 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:04:07.449 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:04:07.449 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.449 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.449 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:04:07.449 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_xinfo": [ 2024-08-30T16:04:07.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.449 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-08-30T16:04:07.449 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:07.450 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:07.450 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:07.450 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:07.450 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:04:07.450 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.911895+0000", 2024-08-30T16:04:07.450 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:04:07.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.450 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-08-30T16:04:07.450 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:07.450 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:07.450 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:07.450 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:07.451 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:04:07.451 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:04.786543+0000", 2024-08-30T16:04:07.451 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:04:07.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.451 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-08-30T16:04:07.451 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:07.451 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:07.451 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:07.451 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:07.451 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:04:07.451 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:04.197952+0000", 2024-08-30T16:04:07.451 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:04:07.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.452 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-08-30T16:04:07.452 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:07.452 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:07.452 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:07.452 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:07.452 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:04:07.452 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.680847+0000", 2024-08-30T16:04:07.452 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:04:07.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.452 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-08-30T16:04:07.452 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:04:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:01.937600+0000", 2024-08-30T16:04:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:04:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-08-30T16:04:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:07.453 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:04:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:02.082761+0000", 2024-08-30T16:04:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:04:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-08-30T16:04:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:04:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.401025+0000", 2024-08-30T16:04:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 74 2024-08-30T16:04:07.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.455 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-08-30T16:04:07.455 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:07.455 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:07.455 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:07.455 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:07.455 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:04:07.455 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.129371+0000", 2024-08-30T16:04:07.455 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:04:07.455 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.455 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:04:07.455 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap": [ 2024-08-30T16:04:07.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.456 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "2.3", 2024-08-30T16:04:07.456 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-08-30T16:04:07.456 INFO:teuthology.orchestra.run.smithi050.stdout: 7, 2024-08-30T16:04:07.456 INFO:teuthology.orchestra.run.smithi050.stdout: 0 2024-08-30T16:04:07.456 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:07.456 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "2.7", 2024-08-30T16:04:07.456 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-08-30T16:04:07.456 INFO:teuthology.orchestra.run.smithi050.stdout: 4, 2024-08-30T16:04:07.456 INFO:teuthology.orchestra.run.smithi050.stdout: 1 2024-08-30T16:04:07.456 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:07.456 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.457 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:04:07.457 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_items": [], 2024-08-30T16:04:07.457 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_primaries": [], 2024-08-30T16:04:07.457 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_temp": [], 2024-08-30T16:04:07.457 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_temp": [], 2024-08-30T16:04:07.457 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklist": {}, 2024-08-30T16:04:07.457 INFO:teuthology.orchestra.run.smithi050.stdout: "range_blocklist": {}, 2024-08-30T16:04:07.457 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profiles": { 2024-08-30T16:04:07.457 INFO:teuthology.orchestra.run.smithi050.stdout: "default": { 2024-08-30T16:04:07.457 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-08-30T16:04:07.457 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-08-30T16:04:07.457 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-08-30T16:04:07.457 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-08-30T16:04:07.457 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van" 2024-08-30T16:04:07.458 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:07.458 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps_queue": [], 2024-08-30T16:04:07.458 INFO:teuthology.orchestra.run.smithi050.stdout: "new_removed_snaps": [], 2024-08-30T16:04:07.458 INFO:teuthology.orchestra.run.smithi050.stdout: "new_purged_snaps": [], 2024-08-30T16:04:07.458 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_node_flags": {}, 2024-08-30T16:04:07.458 INFO:teuthology.orchestra.run.smithi050.stdout: "device_class_flags": {}, 2024-08-30T16:04:07.458 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": { 2024-08-30T16:04:07.458 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_enabled": false, 2024-08-30T16:04:07.458 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_bucket_count": 0, 2024-08-30T16:04:07.458 INFO:teuthology.orchestra.run.smithi050.stdout: "degraded_stretch_mode": 0, 2024-08-30T16:04:07.458 INFO:teuthology.orchestra.run.smithi050.stdout: "recovering_stretch_mode": 0, 2024-08-30T16:04:07.458 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_bucket": 0 2024-08-30T16:04:07.459 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:07.459 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:07.460 INFO:tasks.thrashosds.thrasher:j is {'epoch': 82, 'fsid': '9d72e2e2-0554-42af-8e62-9b1a3a804112', 'created': '2024-08-30T15:58:20.129410+0000', 'modified': '2024-08-30T16:03:50.972329+0000', 'last_up_change': '2024-08-30T16:03:35.238525+0000', 'last_in_change': '2024-08-30T16:03:05.847464+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T15:59:08.355540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-08-30T15:59:21.696796+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 3.9700000286102295, 'score_stable': 3.9700000286102295, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 60, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6812', 'nonce': 1957399494}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6813', 'nonce': 1957399494}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 1957399494}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6814', 'nonce': 1957399494}]}, 'public_addr': '172.21.15.50:6812/1957399494', 'cluster_addr': '172.21.15.50:6813/1957399494', 'heartbeat_back_addr': '172.21.15.50:6815/1957399494', 'heartbeat_front_addr': '172.21.15.50:6814/1957399494', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0fe11a1c-4326-4bfd-a2a4-bff14023f301', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6808', 'nonce': 3208178756}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6809', 'nonce': 3208178756}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 3208178756}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6810', 'nonce': 3208178756}]}, 'public_addr': '172.21.15.50:6808/3208178756', 'cluster_addr': '172.21.15.50:6809/3208178756', 'heartbeat_back_addr': '172.21.15.50:6811/3208178756', 'heartbeat_front_addr': '172.21.15.50:6810/3208178756', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a9d5354b-3600-41ed-80c5-a3439da7aca6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6804', 'nonce': 287018800}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6805', 'nonce': 287018800}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 287018800}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6806', 'nonce': 287018800}]}, 'public_addr': '172.21.15.50:6804/287018800', 'cluster_addr': '172.21.15.50:6805/287018800', 'heartbeat_back_addr': '172.21.15.50:6807/287018800', 'heartbeat_front_addr': '172.21.15.50:6806/287018800', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '78bcc398-a7e4-4efc-9836-1dfb94407c9c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 64, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6800', 'nonce': 1140717714}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6801', 'nonce': 1140717714}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 1140717714}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6802', 'nonce': 1140717714}]}, 'public_addr': '172.21.15.50:6800/1140717714', 'cluster_addr': '172.21.15.50:6801/1140717714', 'heartbeat_back_addr': '172.21.15.50:6803/1140717714', 'heartbeat_front_addr': '172.21.15.50:6802/1140717714', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '53a1060b-7985-4c2c-820e-64b158056113', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 24, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 1468073345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6806', 'nonce': 1468073345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6808', 'nonce': 1468073345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 1468073345}]}, 'public_addr': '172.21.15.94:6805/1468073345', 'cluster_addr': '172.21.15.94:6806/1468073345', 'heartbeat_back_addr': '172.21.15.94:6808/1468073345', 'heartbeat_front_addr': '172.21.15.94:6807/1468073345', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '853a03f7-227d-4f98-9e5c-c29896820746', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 64, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 3876025616}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6810', 'nonce': 3876025616}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6812', 'nonce': 3876025616}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 3876025616}]}, 'public_addr': '172.21.15.94:6809/3876025616', 'cluster_addr': '172.21.15.94:6810/3876025616', 'heartbeat_back_addr': '172.21.15.94:6812/3876025616', 'heartbeat_front_addr': '172.21.15.94:6811/3876025616', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bc978060-bf28-4027-8bba-7d6d8659bfc5', 'up': 0, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 0, 'down_at': 75, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 128519122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6802', 'nonce': 128519122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6804', 'nonce': 128519122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 128519122}]}, 'public_addr': '172.21.15.94:6801/128519122', 'cluster_addr': '172.21.15.94:6802/128519122', 'heartbeat_back_addr': '172.21.15.94:6804/128519122', 'heartbeat_front_addr': '172.21.15.94:6803/128519122', 'state': ['exists']}, {'osd': 7, 'uuid': '45df5c6f-b124-46d1-9efb-62b78bc584ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 68721277}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6814', 'nonce': 68721277}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6816', 'nonce': 68721277}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 68721277}]}, 'public_addr': '172.21.15.94:6813/68721277', 'cluster_addr': '172.21.15.94:6814/68721277', 'heartbeat_back_addr': '172.21.15.94:6816/68721277', 'heartbeat_front_addr': '172.21.15.94:6815/68721277', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.911895+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:04.786543+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:04.197952+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.680847+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:01.937600+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:02.082761+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.401025+0000', 'dead_epoch': 74}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.129371+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.3', 'osds': [7, 0]}, {'pgid': '2.7', 'osds': [4, 1]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T16:04:07.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:04:07.477 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:07.475+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:07.487 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:07.487 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:07.487 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:07.487 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:07.487 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:07.496 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:07.561 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:07.560+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:07.662 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:07.661+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:07.665 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-08-30T16:04:07.669 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:04:07.673 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:07.673 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:07.673 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:07.673 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:04:07.673 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:07.682 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:07.684 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2024-08-30T16:04:07.684 INFO:tasks.mon_recovery:causing some monitor log activity 2024-08-30T16:04:07.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '1 of 30' 2024-08-30T16:04:07.769 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:07.767+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:07.815 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:07.815 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:07.815 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:07.815 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:07.824 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:07.864 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:07.862+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:07.953 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:07.953 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:07.953 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:07.953 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:07.953 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:07.962 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:08.013 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:08.011+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:08.065 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:08.064+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:08.138 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:08.138 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:08.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:08.138 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:08.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.138 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:08.138 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:08.138 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 297.27623852099998, 2024-08-30T16:04:08.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:04:08.139 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.139 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.139 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.139 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.139 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.139 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:04:08.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.139 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:08.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:08.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:04:08.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:04:08.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:04:08.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:04:08.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:04:08.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:04:08.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:04:08.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:04:08.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:04:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:04:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:04:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:04:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:04:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:04:08.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:04:08.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:04:08.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:04:08.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.143 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:04:08.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:04:08.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.143 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:04:08.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:04:08.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:04:08.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:04:08.144 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.144 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.144 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.144 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:08.145 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:08.145 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 296.359822501, 2024-08-30T16:04:08.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:04:08.145 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.145 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.145 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.145 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.145 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.145 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:04:08.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.145 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:08.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:08.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:04:08.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:08.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:04:08.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:04:08.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:04:08.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:04:08.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:04:08.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:04:08.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:04:08.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:04:08.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:04:08.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:04:08.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:04:08.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:04:08.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:04:08.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:04:08.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:04:08.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:04:08.149 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.150 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.150 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.150 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:08.150 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:08.150 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 296.35811794799997, 2024-08-30T16:04:08.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:04:08.150 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.150 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.150 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.150 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.150 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.151 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:04:08.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.151 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:08.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:08.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:04:08.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:04:08.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:04:08.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:04:08.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:04:08.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:04:08.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:04:08.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:04:08.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:04:08.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:04:08.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:04:08.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:04:08.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:04:08.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:04:08.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:04:08.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:04:08.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:04:08.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:04:08.155 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.155 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.155 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.155 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.155 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:08.156 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 295.938664141, 2024-08-30T16:04:08.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:04:08.156 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.156 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.156 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.156 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.156 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.156 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:04:08.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.156 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:08.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:08.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:04:08.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:04:08.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:04:08.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:04:08.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:04:08.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:04:08.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:04:08.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:04:08.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:04:08.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:04:08.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:04:08.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:04:08.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:04:08.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:04:08.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:04:08.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:04:08.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:04:08.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:04:08.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.161 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.161 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.161 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:08.161 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 295.92177596699997, 2024-08-30T16:04:08.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:04:08.161 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.161 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.161 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.161 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.162 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.162 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:04:08.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.162 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:08.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:08.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:04:08.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:04:08.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:04:08.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:04:08.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:04:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:04:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:04:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:04:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:04:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:04:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:04:08.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:04:08.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:04:08.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:04:08.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:04:08.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:04:08.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:04:08.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:04:08.166 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.166 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.166 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.166 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.166 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:08.166 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 295.90877589799999, 2024-08-30T16:04:08.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:04:08.166 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.167 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.167 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.167 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.167 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.167 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:04:08.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.167 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:08.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:08.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:04:08.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:04:08.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:04:08.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:04:08.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:04:08.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:04:08.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:04:08.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:04:08.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:04:08.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:04:08.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:04:08.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:04:08.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:04:08.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:04:08.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:04:08.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:04:08.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:04:08.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:04:08.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.171 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.171 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.171 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:08.171 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 295.90628360599999, 2024-08-30T16:04:08.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:04:08.172 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.172 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.172 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.172 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.172 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.172 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:04:08.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.172 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:08.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:08.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:04:08.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:08.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:04:08.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:04:08.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:04:08.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:04:08.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:04:08.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:04:08.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:04:08.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:04:08.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:04:08.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:04:08.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.176 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.177 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.177 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:08.177 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 295.90552615799999, 2024-08-30T16:04:08.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:04:08.177 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.177 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.177 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:04:08.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.177 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:08.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:08.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:04:08.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:04:08.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:04:08.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:04:08.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:04:08.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:04:08.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:04:08.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:04:08.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:04:08.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:04:08.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:04:08.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:04:08.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:04:08.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:04:08.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:04:08.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:04:08.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:04:08.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:04:08.181 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.182 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.182 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.182 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.182 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:08.182 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 295.904793708, 2024-08-30T16:04:08.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:04:08.182 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.182 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.182 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.183 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.183 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:04:08.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.183 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:08.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:08.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:04:08.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:08.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:04:08.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:04:08.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:04:08.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:04:08.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:04:08.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:04:08.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:04:08.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:04:08.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:04:08.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:04:08.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.186 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.186 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:04:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:04:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:04:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:04:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:04:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:04:08.187 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.188 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.188 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.188 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.188 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:08.188 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 295.90225490900002, 2024-08-30T16:04:08.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:04:08.188 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.188 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.189 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.189 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:04:08.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.189 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:08.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:08.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:04:08.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:08.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:04:08.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:04:08.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:04:08.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:04:08.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:04:08.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:04:08.191 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:04:08.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:04:08.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:04:08.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:04:08.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.192 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.192 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.192 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:04:08.192 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:04:08.192 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:04:08.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:04:08.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.193 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.193 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:04:08.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:04:08.193 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.193 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.193 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.193 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.194 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.194 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:08.194 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 295.90150690799999, 2024-08-30T16:04:08.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:04:08.194 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.194 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.194 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.194 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.194 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.194 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:04:08.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.194 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:08.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:08.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:04:08.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:04:08.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:04:08.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:04:08.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:04:08.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:04:08.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:04:08.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:04:08.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:04:08.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:04:08.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:04:08.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:04:08.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:04:08.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:04:08.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:04:08.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:04:08.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:04:08.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:04:08.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.199 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.199 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.199 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.199 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:08.199 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 295.12939043599999, 2024-08-30T16:04:08.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:04:08.199 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.199 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.199 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.199 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.199 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.200 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:04:08.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.200 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:08.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.200 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.200 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:08.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:04:08.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:08.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:04:08.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:04:08.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:04:08.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:04:08.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:04:08.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:04:08.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:04:08.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:04:08.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:04:08.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:04:08.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:04:08.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:04:08.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:04:08.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:04:08.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:04:08.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:04:08.204 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.204 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.204 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.205 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.205 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:08.205 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 295.12733329000002, 2024-08-30T16:04:08.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:04:08.205 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.205 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.205 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.205 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.205 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.205 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:04:08.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.205 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.206 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:08.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:08.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:04:08.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:08.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:04:08.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:04:08.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:04:08.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:04:08.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:04:08.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:04:08.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:04:08.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:04:08.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:04:08.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:04:08.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:04:08.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:04:08.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:04:08.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:04:08.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.212 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:04:08.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:04:08.212 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.212 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.212 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.213 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.213 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:08.213 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 294.86438879000002, 2024-08-30T16:04:08.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:04:08.213 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.213 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.213 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.213 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.213 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.213 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:04:08.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.213 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.213 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.213 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:08.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:08.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:04:08.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:04:08.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:04:08.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:04:08.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:04:08.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:04:08.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:04:08.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:04:08.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:04:08.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:04:08.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:04:08.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:04:08.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:04:08.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:04:08.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:04:08.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:04:08.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:04:08.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:04:08.218 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.218 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.218 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.218 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.218 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:08.218 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 294.86329728700002, 2024-08-30T16:04:08.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:04:08.218 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.218 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.218 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.218 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.218 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.219 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:04:08.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.219 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:08.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:08.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:04:08.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:08.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:04:08.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:04:08.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:04:08.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:04:08.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:04:08.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:04:08.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:04:08.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:04:08.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:04:08.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:04:08.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:04:08.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:04:08.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:04:08.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:04:08.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.224 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.224 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:04:08.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:04:08.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.224 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.224 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.224 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:08.224 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 294.86254770199997, 2024-08-30T16:04:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:04:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:04:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:08.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:04:08.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:08.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:04:08.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:04:08.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:04:08.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:04:08.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:04:08.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:04:08.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:04:08.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:04:08.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:04:08.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:04:08.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:04:08.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:04:08.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:04:08.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:04:08.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:04:08.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:04:08.230 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.230 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.230 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.230 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.230 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:08.230 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 294.82881616999998, 2024-08-30T16:04:08.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:04:08.230 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.230 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.231 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.231 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:04:08.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.231 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:08.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:08.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:04:08.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:04:08.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:04:08.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:04:08.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:04:08.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:04:08.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:04:08.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:04:08.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:04:08.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:04:08.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:04:08.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:04:08.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:04:08.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:04:08.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:04:08.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:04:08.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:04:08.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:04:08.235 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.236 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.236 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.236 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.236 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:08.236 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 294.82829016199997, 2024-08-30T16:04:08.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:04:08.236 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.236 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.236 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.236 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.237 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.237 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:04:08.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.237 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:08.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:08.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:04:08.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:08.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:04:08.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:04:08.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:04:08.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:04:08.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:04:08.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:04:08.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:04:08.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:04:08.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:04:08.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:04:08.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:04:08.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:04:08.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:04:08.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:04:08.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:04:08.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:04:08.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.241 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.242 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:08.242 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:08.242 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 294.82783459900003, 2024-08-30T16:04:08.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:04:08.242 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.242 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.242 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.242 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.242 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.242 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:04:08.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.242 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:08.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:08.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:04:08.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:04:08.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:04:08.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:04:08.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:04:08.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:04:08.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:04:08.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:04:08.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:04:08.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:04:08.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:04:08.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:04:08.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:04:08.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:04:08.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:04:08.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:04:08.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:04:08.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:04:08.247 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.247 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.247 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.247 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:04:08.247 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:08.247 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 290.15731928299999, 2024-08-30T16:04:08.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:04:08.247 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.247 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:08.248 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.248 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:08.248 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:08.248 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:04:08.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.248 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:08.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:08.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:04:08.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:08.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:04:08.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:04:08.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:04:08.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:04:08.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:04:08.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:04:08.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:04:08.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:04:08.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:04:08.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:04:08.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:08.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:04:08.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:04:08.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:08.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:04:08.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:04:08.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:04:08.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:04:08.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.252 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.252 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.253 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:08.254 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:04:08.254 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:04:08.256 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:08.257 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-08-30T16:04:07.919049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":397859,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":138164,"kb_used_data":3036,"kb_used_omap":7,"kb_used_meta":135032,"kb_avail":471721036,"statfs":{"total":483183820800,"available":483042340864,"internally_reserved":0,"allocated":3108864,"data_stored":2310698,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":7967,"internal_metadata":138272993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001820"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":71,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-08-30T16:03:51.025967+0000","last_change":"2024-08-30T15:59:27.646756+0000","last_active":"2024-08-30T16:03:51.025967+0000","last_peered":"2024-08-30T16:03:51.025967+0000","last_clean":"2024-08-30T16:03:51.025967+0000","last_became_active":"2024-08-30T15:59:25.342321+0000","last_became_peered":"2024-08-30T15:59:25.342321+0000","last_unstale":"2024-08-30T16:03:51.025967+0000","last_undegraded":"2024-08-30T16:03:51.025967+0000","last_fullsized":"2024-08-30T16:03:51.025967+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:53:42.927368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.116914037,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-08-30T16:03:51.032261+0000","last_change":"2024-08-30T16:02:40.613055+0000","last_active":"2024-08-30T16:03:51.032261+0000","last_peered":"2024-08-30T16:03:51.032261+0000","last_clean":"2024-08-30T16:03:51.032261+0000","last_became_active":"2024-08-30T16:02:40.576990+0000","last_became_peered":"2024-08-30T16:02:40.576990+0000","last_unstale":"2024-08-30T16:03:51.032261+0000","last_undegraded":"2024-08-30T16:03:51.032261+0000","last_fullsized":"2024-08-30T16:03:51.032261+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T01:33:27.937690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":79,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-08-30T16:03:51.023420+0000","last_change":"2024-08-30T16:02:40.454608+0000","last_active":"2024-08-30T16:03:51.023420+0000","last_peered":"2024-08-30T16:03:51.023420+0000","last_clean":"2024-08-30T16:03:51.023420+0000","last_became_active":"2024-08-30T16:02:40.448507+0000","last_became_peered":"2024-08-30T16:02:40.448507+0000","last_unstale":"2024-08-30T16:03:51.023420+0000","last_undegraded":"2024-08-30T16:03:51.023420+0000","last_fullsized":"2024-08-30T16:03:51.023420+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T00:30:34.065701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":85,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-08-30T16:03:51.013915+0000","last_change":"2024-08-30T16:03:08.888511+0000","last_active":"2024-08-30T16:03:51.013915+0000","last_peered":"2024-08-30T16:03:51.013915+0000","last_clean":"2024-08-30T16:03:51.013915+0000","last_became_active":"2024-08-30T16:03:08.882505+0000","last_became_peered":"2024-08-30T16:03:08.882505+0000","last_unstale":"2024-08-30T16:03:51.013915+0000","last_undegraded":"2024-08-30T16:03:51.013915+0000","last_fullsized":"2024-08-30T16:03:51.013915+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:06:24.414887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":85,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-08-30T16:03:51.012460+0000","last_change":"2024-08-30T16:01:01.435839+0000","last_active":"2024-08-30T16:03:51.012460+0000","last_peered":"2024-08-30T16:03:51.012460+0000","last_clean":"2024-08-30T16:03:51.012460+0000","last_became_active":"2024-08-30T16:01:00.974833+0000","last_became_peered":"2024-08-30T16:01:00.974833+0000","last_unstale":"2024-08-30T16:03:51.012460+0000","last_undegraded":"2024-08-30T16:03:51.012460+0000","last_fullsized":"2024-08-30T16:03:51.012460+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:09:38.575833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":69,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-08-30T16:03:50.486978+0000","last_change":"2024-08-30T15:59:27.825532+0000","last_active":"2024-08-30T16:03:50.486978+0000","last_peered":"2024-08-30T16:03:50.486978+0000","last_clean":"2024-08-30T16:03:50.486978+0000","last_became_active":"2024-08-30T15:59:25.469390+0000","last_became_peered":"2024-08-30T15:59:25.469390+0000","last_unstale":"2024-08-30T16:03:50.486978+0000","last_undegraded":"2024-08-30T16:03:50.486978+0000","last_fullsized":"2024-08-30T16:03:50.486978+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T17:46:39.688911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":71,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-08-30T16:03:51.010435+0000","last_change":"2024-08-30T15:59:27.583986+0000","last_active":"2024-08-30T16:03:51.010435+0000","last_peered":"2024-08-30T16:03:51.010435+0000","last_clean":"2024-08-30T16:03:51.010435+0000","last_became_active":"2024-08-30T15:59:24.528536+0000","last_became_peered":"2024-08-30T15:59:24.528536+0000","last_unstale":"2024-08-30T16:03:51.010435+0000","last_undegraded":"2024-08-30T16:03:51.010435+0000","last_fullsized":"2024-08-30T16:03:51.010435+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:18:40.602710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":2,"reported_epoch":82,"state":"peering","last_fresh":"2024-08-30T16:03:08.214899+0000","last_change":"2024-08-30T16:03:07.516224+0000","last_active":"2024-08-30T16:03:07.512274+0000","last_peered":"2024-08-30T16:02:49.019094+0000","last_clean":"2024-08-30T16:02:49.019094+0000","last_became_active":"2024-08-30T16:01:00.806973+0000","last_became_peered":"2024-08-30T16:01:00.806973+0000","last_unstale":"2024-08-30T16:03:08.214899+0000","last_undegraded":"2024-08-30T16:03:08.214899+0000","last_fullsized":"2024-08-30T16:03:08.214899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:13:37.016725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":72,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-08-30T16:03:51.004087+0000","last_change":"2024-08-30T15:59:21.623783+0000","last_active":"2024-08-30T16:03:51.004087+0000","last_peered":"2024-08-30T16:03:51.004087+0000","last_clean":"2024-08-30T16:03:51.004087+0000","last_became_active":"2024-08-30T15:59:20.848924+0000","last_became_peered":"2024-08-30T15:59:20.848924+0000","last_unstale":"2024-08-30T16:03:51.004087+0000","last_undegraded":"2024-08-30T16:03:51.004087+0000","last_fullsized":"2024-08-30T16:03:51.004087+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:14:52.575421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":795680,"data_compressed":1468,"data_compressed_allocated":49152,"data_compressed_original":782336,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280578,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":643072,"data_stored":700855,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":12.510999999999999,"5min":12.433,"15min":13.757999999999999},"min":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"max":{"1min":51.027000000000001,"5min":57.177999999999997,"15min":57.177999999999997},"last":26.73},{"interface":"front","average":{"1min":13.460000000000001,"5min":12.444000000000001,"15min":13.041},"min":{"1min":6.883,"5min":5.0510000000000002,"15min":5.0510000000000002},"max":{"1min":54.682000000000002,"5min":58.024999999999999,"15min":58.024999999999999},"last":24.952999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":9.5579999999999998,"5min":10.762,"15min":10.539999999999999},"min":{"1min":5.5780000000000003,"5min":5.5780000000000003,"15min":5.5780000000000003},"max":{"1min":20.256,"5min":73.399000000000001,"15min":73.399000000000001},"last":13.510999999999999},{"interface":"front","average":{"1min":14.210000000000001,"5min":14.776999999999999,"15min":14.977},"min":{"1min":8.4920000000000009,"5min":6.9139999999999997,"15min":6.9139999999999997},"max":{"1min":26.715,"5min":75.120000000000005,"15min":75.120000000000005},"last":6.0049999999999999}]},{"osd":3,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":17.975999999999999,"5min":16.571999999999999,"15min":17.300000000000001},"min":{"1min":8.4849999999999994,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":66.869,"5min":66.869,"15min":66.869},"last":12.140000000000001},{"interface":"front","average":{"1min":13.228,"5min":14.532,"15min":14.795999999999999},"min":{"1min":7.9930000000000003,"5min":7.6619999999999999,"15min":7.6619999999999999},"max":{"1min":29.524999999999999,"5min":104.38200000000001,"15min":104.38200000000001},"last":7.6520000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":17.800000000000001,"5min":15.215,"15min":14.923},"min":{"1min":6.7279999999999998,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":66.340999999999994,"5min":66.340999999999994,"15min":66.340999999999994},"last":9.3160000000000007},{"interface":"front","average":{"1min":13.164999999999999,"5min":13.124000000000001,"15min":13.084},"min":{"1min":7.8970000000000002,"5min":7.0039999999999996,"15min":7.0039999999999996},"max":{"1min":20.760999999999999,"5min":29.818999999999999,"15min":29.818999999999999},"last":7.0119999999999996}]},{"osd":5,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":14.279999999999999,"5min":16.236999999999998,"15min":16.689},"min":{"1min":7.8920000000000003,"5min":7.8920000000000003,"15min":7.8920000000000003},"max":{"1min":25.75,"5min":55.784999999999997,"15min":55.784999999999997},"last":14.256},{"interface":"front","average":{"1min":18.994,"5min":18.856999999999999,"15min":19.475999999999999},"min":{"1min":11.208,"5min":9.5800000000000001,"15min":9.5800000000000001},"max":{"1min":68.004999999999995,"5min":68.004999999999995,"15min":68.004999999999995},"last":10.504}]},{"osd":6,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":15.708,"5min":16.609000000000002,"15min":17.382000000000001},"min":{"1min":10.180999999999999,"5min":9.1929999999999996,"15min":9.1929999999999996},"max":{"1min":27.353999999999999,"5min":58.201999999999998,"15min":58.201999999999998},"last":18.898},{"interface":"front","average":{"1min":16.324999999999999,"5min":16.748000000000001,"15min":17.001000000000001},"min":{"1min":9.6980000000000004,"5min":8.8390000000000004,"15min":8.8390000000000004},"max":{"1min":26.257999999999999,"5min":37.253,"15min":37.253},"last":12.787000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":21.238,"5min":18.891999999999999,"15min":19.809999999999999},"min":{"1min":8.9770000000000003,"5min":8.2050000000000001,"15min":8.2050000000000001},"max":{"1min":68.950000000000003,"5min":68.950000000000003,"15min":68.950000000000003},"last":11.308},{"interface":"front","average":{"1min":18.027999999999999,"5min":18.108000000000001,"15min":19.058},"min":{"1min":8.5850000000000009,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":31.367000000000001,"5min":59.079000000000001,"15min":59.079000000000001},"last":8.8089999999999993}]}]},{"osd":7,"up_from":21,"seq":90194313281,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":302996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":8.4659999999999993,"5min":9.2919999999999998,"15min":9.0730000000000004},"min":{"1min":4.8079999999999998,"5min":4.7169999999999996,"15min":4.7169999999999996},"max":{"1min":16.690000000000001,"5min":30.600999999999999,"15min":30.600999999999999},"last":4.8029999999999999},{"interface":"front","average":{"1min":10.06,"5min":9.4169999999999998,"15min":9.2210000000000001},"min":{"1min":4.9779999999999998,"5min":4.5430000000000001,"15min":4.5430000000000001},"max":{"1min":26.928000000000001,"5min":28.077000000000002,"15min":28.077000000000002},"last":8.1760000000000002}]},{"osd":1,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":11.928000000000001,"5min":12.59,"15min":11.42},"min":{"1min":5.7270000000000003,"5min":5.7270000000000003,"15min":5.7270000000000003},"max":{"1min":23.419,"5min":83.581000000000003,"15min":83.581000000000003},"last":6.6509999999999998},{"interface":"front","average":{"1min":11.988,"5min":12.587999999999999,"15min":12.467000000000001},"min":{"1min":6.3719999999999999,"5min":6.3719999999999999,"15min":6.3719999999999999},"max":{"1min":27.940000000000001,"5min":81.722999999999999,"15min":81.722999999999999},"last":6.9809999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.329000000000001,"5min":13.84,"15min":13.672000000000001},"min":{"1min":6.6559999999999997,"5min":6.5970000000000004,"15min":6.5970000000000004},"max":{"1min":42.506,"5min":42.506,"15min":42.506},"last":9.8130000000000006},{"interface":"front","average":{"1min":15.164,"5min":15.433,"15min":16.823},"min":{"1min":7.673,"5min":6.9800000000000004,"15min":6.9800000000000004},"max":{"1min":43.484999999999999,"5min":43.484999999999999,"15min":43.484999999999999},"last":10.331}]},{"osd":3,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":15.202999999999999,"5min":16.597999999999999,"15min":16.596},"min":{"1min":10.531000000000001,"5min":8.8420000000000005,"15min":8.8420000000000005},"max":{"1min":35.677999999999997,"5min":65.739999999999995,"15min":65.739999999999995},"last":14.005000000000001},{"interface":"front","average":{"1min":11.984999999999999,"5min":13.198,"15min":12.842000000000001},"min":{"1min":6.2119999999999997,"5min":6.2119999999999997,"15min":6.2119999999999997},"max":{"1min":36.258000000000003,"5min":64.980000000000004,"15min":64.980000000000004},"last":9.0449999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":14.795,"5min":13.742000000000001,"15min":12.800000000000001},"min":{"1min":8.3670000000000009,"5min":7.0339999999999998,"15min":7.0339999999999998},"max":{"1min":24.486999999999998,"5min":37.746000000000002,"15min":37.746000000000002},"last":11.542999999999999},{"interface":"front","average":{"1min":14.532999999999999,"5min":16.396999999999998,"15min":17.324999999999999},"min":{"1min":10.377000000000001,"5min":9.2070000000000007,"15min":9.2070000000000007},"max":{"1min":22.492000000000001,"5min":37.393000000000001,"15min":37.393000000000001},"last":12.789}]},{"osd":5,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":12.417999999999999,"5min":11.829000000000001,"15min":11.496},"min":{"1min":7.6440000000000001,"5min":6.2309999999999999,"15min":6.2309999999999999},"max":{"1min":23.266999999999999,"5min":29.754999999999999,"15min":29.754999999999999},"last":13.337999999999999},{"interface":"front","average":{"1min":15.071,"5min":16.338999999999999,"15min":16.789999999999999},"min":{"1min":8.407,"5min":8.407,"15min":8.407},"max":{"1min":24.846,"5min":38.618000000000002,"15min":38.618000000000002},"last":10.978999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.326000000000001,"5min":13.771000000000001,"15min":13.208},"min":{"1min":8.0060000000000002,"5min":6.3230000000000004,"15min":6.3230000000000004},"max":{"1min":19.231000000000002,"5min":42.829999999999998,"15min":42.829999999999998},"last":11.368},{"interface":"front","average":{"1min":15.842000000000001,"5min":17.606000000000002,"15min":18.088999999999999},"min":{"1min":11.712,"5min":10.749000000000001,"15min":10.749000000000001},"max":{"1min":22.695,"5min":44.491999999999997,"15min":44.491999999999997},"last":14.314}]}]},{"osd":6,"up_from":0,"seq":90194313274,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":21,"seq":90194313280,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":303015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":10.545,"5min":10.287000000000001,"15min":11.180999999999999},"min":{"1min":5.0599999999999996,"5min":4.29,"15min":4.29},"max":{"1min":27.635000000000002,"5min":50.500999999999998,"15min":50.500999999999998},"last":7.7619999999999996},{"interface":"front","average":{"1min":11.278,"5min":12.206,"15min":14.045999999999999},"min":{"1min":5.9470000000000001,"5min":5.1360000000000001,"15min":5.1360000000000001},"max":{"1min":36.720999999999997,"5min":51.965000000000003,"15min":51.965000000000003},"last":5.3890000000000002}]},{"osd":1,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":14.144,"5min":15.917,"15min":18.513999999999999},"min":{"1min":8.2029999999999994,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":31.433,"5min":137.56700000000001,"15min":137.56700000000001},"last":14.083},{"interface":"front","average":{"1min":13.198,"5min":14.635,"15min":16.535},"min":{"1min":6.4779999999999998,"5min":5.0060000000000002,"15min":5.0060000000000002},"max":{"1min":24.753,"5min":136.321,"15min":136.321},"last":11.605}]},{"osd":2,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":12.728999999999999,"5min":14.698,"15min":16.170000000000002},"min":{"1min":7.2480000000000002,"5min":7.2480000000000002,"15min":7.2480000000000002},"max":{"1min":26.210000000000001,"5min":73.831000000000003,"15min":73.831000000000003},"last":6.3470000000000004},{"interface":"front","average":{"1min":10.778,"5min":13.547000000000001,"15min":15.545},"min":{"1min":6.7050000000000001,"5min":6.7050000000000001,"15min":6.7050000000000001},"max":{"1min":19.884,"5min":71.831999999999994,"15min":71.831999999999994},"last":10.984999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":14.756,"5min":21.741,"15min":28.48},"min":{"1min":7.7949999999999999,"5min":7.7949999999999999,"15min":7.7949999999999999},"max":{"1min":32.465000000000003,"5min":342.34500000000003,"15min":342.34500000000003},"last":8.4049999999999994},{"interface":"front","average":{"1min":16.163,"5min":17.98,"15min":20.091000000000001},"min":{"1min":10.538,"5min":8.0299999999999994,"15min":8.0299999999999994},"max":{"1min":29.649999999999999,"5min":127.857,"15min":127.857},"last":7.218}]},{"osd":5,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.073,"5min":15.792999999999999,"15min":16},"min":{"1min":7.5250000000000004,"5min":7.2039999999999997,"15min":7.2039999999999997},"max":{"1min":55.005000000000003,"5min":58.715000000000003,"15min":58.715000000000003},"last":10.427},{"interface":"front","average":{"1min":19.274000000000001,"5min":18.774000000000001,"15min":19.256},"min":{"1min":7.5869999999999997,"5min":7.2370000000000001,"15min":7.2370000000000001},"max":{"1min":54.058,"5min":58.212000000000003,"15min":58.212000000000003},"last":9.5489999999999995}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.295000000000002,"5min":23.492000000000001,"15min":30.053999999999998},"min":{"1min":11.096,"5min":8.6630000000000003,"15min":8.6630000000000003},"max":{"1min":25.298999999999999,"5min":388.96100000000001,"15min":388.96100000000001},"last":15.696},{"interface":"front","average":{"1min":13.961,"5min":24.18,"15min":32.835999999999999},"min":{"1min":9.4079999999999995,"5min":8.5519999999999996,"15min":8.5519999999999996},"max":{"1min":20.439,"5min":448.69200000000001,"15min":448.69200000000001},"last":13.44}]},{"osd":7,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":17.707999999999998,"5min":21.858000000000001,"15min":25.350999999999999},"min":{"1min":10.74,"5min":9.1259999999999994,"15min":9.1259999999999994},"max":{"1min":33.82,"5min":177.596,"15min":177.596},"last":13.35},{"interface":"front","average":{"1min":17.922000000000001,"5min":22.812000000000001,"15min":26.838000000000001},"min":{"1min":11.428000000000001,"5min":9.6270000000000007,"15min":9.6270000000000007},"max":{"1min":28.056999999999999,"5min":231.03800000000001,"15min":231.03800000000001},"last":12.117000000000001}]}]},{"osd":4,"up_from":0,"seq":85899345985,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":68719476799,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":15,"seq":64424509502,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":606208,"data_stored":302996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":10.943,"5min":10.377000000000001,"15min":9.9009999999999998},"min":{"1min":5.3259999999999996,"5min":5.0019999999999998,"15min":5.0019999999999998},"max":{"1min":21.992999999999999,"5min":34.015000000000001,"15min":34.015000000000001},"last":11.02},{"interface":"front","average":{"1min":12.393000000000001,"5min":11.308999999999999,"15min":11.339},"min":{"1min":7.0629999999999997,"5min":4.7210000000000001,"15min":4.7210000000000001},"max":{"1min":29.138000000000002,"5min":36.636000000000003,"15min":36.636000000000003},"last":8.4789999999999992}]},{"osd":2,"last update":"Fri Aug 30 16:03:13 2024","interfaces":[{"interface":"back","average":{"1min":11.630000000000001,"5min":11.348000000000001,"15min":11.256},"min":{"1min":6.1109999999999998,"5min":3.3370000000000002,"15min":3.3370000000000002},"max":{"1min":21.43,"5min":46.768999999999998,"15min":46.768999999999998},"last":6.1840000000000002},{"interface":"front","average":{"1min":10.179,"5min":13.541,"15min":14.135},"min":{"1min":7.5209999999999999,"5min":5.702,"15min":5.702},"max":{"1min":19.795000000000002,"5min":43.921999999999997,"15min":43.921999999999997},"last":11.768000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":14.497999999999999,"5min":18.655999999999999,"15min":23.094999999999999},"min":{"1min":7.4530000000000003,"5min":6.2839999999999998,"15min":6.2839999999999998},"max":{"1min":36.070999999999998,"5min":200.155,"15min":200.155},"last":9.484},{"interface":"front","average":{"1min":18.696999999999999,"5min":23.181999999999999,"15min":28.526},"min":{"1min":11.661,"5min":7.0940000000000003,"15min":7.0940000000000003},"max":{"1min":37.436,"5min":267.03100000000001,"15min":267.03100000000001},"last":13.271000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":14.69,"5min":14.949999999999999,"15min":15.651999999999999},"min":{"1min":7.4329999999999998,"5min":6.6429999999999998,"15min":6.6429999999999998},"max":{"1min":31.611999999999998,"5min":56.200000000000003,"15min":56.200000000000003},"last":7.2590000000000003},{"interface":"front","average":{"1min":17.827000000000002,"5min":16.105,"15min":15.862},"min":{"1min":11.4,"5min":8.609,"15min":8.609},"max":{"1min":46.875999999999998,"5min":48.534999999999997,"15min":48.534999999999997},"last":9.9879999999999995}]},{"osd":5,"last update":"Fri Aug 30 16:03:13 2024","interfaces":[{"interface":"back","average":{"1min":15.835000000000001,"5min":15.973000000000001,"15min":16.18},"min":{"1min":11.196,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":35.521000000000001,"5min":35.521000000000001,"15min":35.521000000000001},"last":7.8609999999999998},{"interface":"front","average":{"1min":17.591000000000001,"5min":17.614000000000001,"15min":17.827999999999999},"min":{"1min":9.2880000000000003,"5min":4.75,"15min":4.75},"max":{"1min":36.832000000000001,"5min":49.396000000000001,"15min":49.396000000000001},"last":14.823}]},{"osd":6,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":13.808999999999999,"5min":14.292,"15min":14.385999999999999},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":39.460999999999999,"5min":59.328000000000003,"15min":59.328000000000003},"last":23.652999999999999},{"interface":"front","average":{"1min":19.225000000000001,"5min":18.331,"15min":18.533999999999999},"min":{"1min":10.781000000000001,"5min":10.005000000000001,"15min":10.005000000000001},"max":{"1min":46.237000000000002,"5min":58.567,"15min":58.567},"last":38.289000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":22.013000000000002,"5min":20.611000000000001,"15min":21.581},"min":{"1min":14.118,"5min":9.1180000000000003,"15min":9.1180000000000003},"max":{"1min":42.585000000000001,"5min":59.515999999999998,"15min":59.515999999999998},"last":13.984999999999999},{"interface":"front","average":{"1min":13.846,"5min":14.859999999999999,"15min":15.468},"min":{"1min":9.3550000000000004,"5min":7.1619999999999999,"15min":7.1619999999999999},"max":{"1min":27.023,"5min":58.807000000000002,"15min":58.807000000000002},"last":12.308}]}]},{"osd":5,"up_from":13,"seq":55834574910,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":643072,"data_stored":700836,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:03:17 2024","interfaces":[{"interface":"back","average":{"1min":10.364000000000001,"5min":11.343,"15min":13.547000000000001},"min":{"1min":4.1299999999999999,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":53.558999999999997,"5min":111.313,"15min":111.313},"last":4.9649999999999999},{"interface":"front","average":{"1min":11.138999999999999,"5min":12.304,"15min":14.882},"min":{"1min":5.4809999999999999,"5min":4.7750000000000004,"15min":4.7750000000000004},"max":{"1min":52.691000000000003,"5min":107.937,"15min":107.937},"last":13.186}]},{"osd":1,"last update":"Fri Aug 30 16:03:29 2024","interfaces":[{"interface":"back","average":{"1min":13.371,"5min":12.039,"15min":13.057},"min":{"1min":5.8719999999999999,"5min":4.657,"15min":4.657},"max":{"1min":34.545999999999999,"5min":46.988,"15min":46.988},"last":12.599},{"interface":"front","average":{"1min":15.677,"5min":14.946,"15min":15.869999999999999},"min":{"1min":6.7350000000000003,"5min":6.7350000000000003,"15min":6.7350000000000003},"max":{"1min":33.203000000000003,"5min":47.661000000000001,"15min":47.661000000000001},"last":6.4710000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:03:17 2024","interfaces":[{"interface":"back","average":{"1min":16.106000000000002,"5min":17.030000000000001,"15min":18.556000000000001},"min":{"1min":9.6470000000000002,"5min":4.7220000000000004,"15min":4.7220000000000004},"max":{"1min":41.549999999999997,"5min":84.317999999999998,"15min":84.317999999999998},"last":7.2720000000000002},{"interface":"front","average":{"1min":14.890000000000001,"5min":15.475,"15min":16.731000000000002},"min":{"1min":6.9530000000000003,"5min":6.0279999999999996,"15min":6.0279999999999996},"max":{"1min":43.314999999999998,"5min":99.884,"15min":99.884},"last":14.295999999999999}]},{"osd":3,"last update":"Fri Aug 30 16:03:26 2024","interfaces":[{"interface":"back","average":{"1min":15.500999999999999,"5min":17.43,"15min":17.207999999999998},"min":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":6.2149999999999999},"max":{"1min":28.585000000000001,"5min":117.346,"15min":117.346},"last":9.4480000000000004},{"interface":"front","average":{"1min":13.127000000000001,"5min":16.134,"15min":16.379999999999999},"min":{"1min":7.1719999999999997,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":20.510000000000002,"5min":119.23099999999999,"15min":119.23099999999999},"last":15.054}]},{"osd":4,"last update":"Fri Aug 30 16:03:26 2024","interfaces":[{"interface":"back","average":{"1min":16.254000000000001,"5min":17.491,"15min":17.895},"min":{"1min":7.9870000000000001,"5min":7.9870000000000001,"15min":7.9870000000000001},"max":{"1min":28.026,"5min":38.095999999999997,"15min":38.095999999999997},"last":10.134},{"interface":"front","average":{"1min":14.409000000000001,"5min":16.515000000000001,"15min":17.207999999999998},"min":{"1min":7.5030000000000001,"5min":7.5030000000000001,"15min":7.5030000000000001},"max":{"1min":31.367000000000001,"5min":40.649000000000001,"15min":40.649000000000001},"last":15.778}]},{"osd":6,"last update":"Fri Aug 30 16:03:26 2024","interfaces":[{"interface":"back","average":{"1min":9.516,"5min":10.686,"15min":11.025},"min":{"1min":6.7809999999999997,"5min":6.367,"15min":6.367},"max":{"1min":15.27,"5min":26.344999999999999,"15min":26.344999999999999},"last":8.5960000000000001},{"interface":"front","average":{"1min":19.596,"5min":17.696999999999999,"15min":17.396000000000001},"min":{"1min":11.433,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":58.052999999999997,"5min":58.052999999999997,"15min":58.052999999999997},"last":13.423}]},{"osd":7,"last update":"Fri Aug 30 16:03:26 2024","interfaces":[{"interface":"back","average":{"1min":14.516999999999999,"5min":16.550999999999998,"15min":17.843},"min":{"1min":10.054,"5min":9.0909999999999993,"15min":9.0909999999999993},"max":{"1min":26.913,"5min":43.604999999999997,"15min":43.604999999999997},"last":17.614000000000001},{"interface":"front","average":{"1min":11.013999999999999,"5min":14.605,"15min":17.475999999999999},"min":{"1min":6.3810000000000002,"5min":6.3810000000000002,"15min":6.3810000000000002},"max":{"1min":15.276,"5min":88.278000000000006,"15min":88.278000000000006},"last":8.3040000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:04:08.258 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [7, 0] 2024-08-30T16:04:08.258 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.7', '7', '0'] 2024-08-30T16:04:08.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.7 7 0 2024-08-30T16:04:08.259 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:08.253+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:08.356 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:08.354+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:08.387 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:08.387 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:08.387 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:08.398 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:08.457 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:08.455+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:08.535 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:08.535 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:08.535 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:08.535 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:08.535 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:08.545 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:08.589 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:08.587+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:08.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:04:08.680 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:08.678+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:08.680 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:08.680 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:08.680 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:08.680 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:08.680 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.681 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:08.681 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:08.681 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 280.29466471799998, 2024-08-30T16:04:08.681 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:04:08.681 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.681 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:08.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:08.681 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:08.681 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:04:08.681 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.681 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:08.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:08.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.682 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.682 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.682 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.682 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:04:08.682 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:04:08.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:04:08.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:04:08.683 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.683 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.683 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.683 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:04:08.683 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:04:08.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.684 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:04:08.684 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:04:08.684 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.684 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.684 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:04:08.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:04:08.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.685 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.685 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:04:08.685 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:04:08.685 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.685 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.685 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.685 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.685 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.686 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:04:08.686 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:08.686 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 160.181668602, 2024-08-30T16:04:08.686 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:04:08.686 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.686 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:08.686 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.686 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:08.687 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:08.687 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:04:08.687 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.687 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.687 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.688 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:08.688 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.688 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.688 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.688 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:08.689 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.689 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.689 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.689 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.689 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:04:08.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:08.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.690 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.690 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.690 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:04:08.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:04:08.690 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:04:08.691 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:04:08.691 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.691 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.691 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.691 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:04:08.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:04:08.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.692 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.692 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.692 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:04:08.692 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:04:08.692 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.693 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.693 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:04:08.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:04:08.693 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.693 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.694 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.694 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-08-30T16:04:08.694 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:08.694 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 40.173249231, 2024-08-30T16:04:08.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037103309999999999, 2024-08-30T16:04:08.695 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.695 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:08.695 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:08.695 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:08.695 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:08.695 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 10 2024-08-30T16:04:08.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.696 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:08.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:08.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496549+0000", 2024-08-30T16:04:08.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:08.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496682+0000", 2024-08-30T16:04:08.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013371699999999999 2024-08-30T16:04:08.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496889+0000", 2024-08-30T16:04:08.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00020673600000000001 2024-08-30T16:04:08.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.497052+0000", 2024-08-30T16:04:08.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000162578 2024-08-30T16:04:08.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.498833+0000", 2024-08-30T16:04:08.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017810530000000001 2024-08-30T16:04:08.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.500282+0000", 2024-08-30T16:04:08.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001449437 2024-08-30T16:04:08.699 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.699 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.699 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.699 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(2.3 83/82 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 25'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@25'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:08.699 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:08.699 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.79220520000000005, 2024-08-30T16:04:08.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088711689999999999, 2024-08-30T16:04:08.700 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.700 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:08.700 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:08.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:08.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877588+0000", 2024-08-30T16:04:08.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999723 2024-08-30T16:04:08.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877978+0000", 2024-08-30T16:04:08.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038947500000000002 2024-08-30T16:04:08.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.878456+0000", 2024-08-30T16:04:08.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047795699999999999 2024-08-30T16:04:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.878946+0000", 2024-08-30T16:04:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000490235 2024-08-30T16:04:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.880512+0000", 2024-08-30T16:04:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001565484 2024-08-30T16:04:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:08.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.880749+0000", 2024-08-30T16:04:08.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000237712 2024-08-30T16:04:08.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.886487+0000", 2024-08-30T16:04:08.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057377349999999999 2024-08-30T16:04:08.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.703 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.703 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(2.3 83/82 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-08-30T16:04:08.704 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:08.704 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 0.64479788999999998, 2024-08-30T16:04:08.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062707819999999999, 2024-08-30T16:04:08.704 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:08.704 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:08.704 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:08.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:08.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:08.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:08.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:08.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:08.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:08.705 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025012+0000", 2024-08-30T16:04:08.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:04:08.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.705 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:08.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025103+0000", 2024-08-30T16:04:08.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0877000000000004e-05 2024-08-30T16:04:08.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:08.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025231+0000", 2024-08-30T16:04:08.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012737499999999999 2024-08-30T16:04:08.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:08.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025355+0000", 2024-08-30T16:04:08.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124194 2024-08-30T16:04:08.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:08.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.026750+0000", 2024-08-30T16:04:08.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001394922 2024-08-30T16:04:08.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:08.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:08.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:08.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.031294+0000", 2024-08-30T16:04:08.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045441270000000002 2024-08-30T16:04:08.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.707 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:08.707 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:08.707 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:08.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:08.759 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:08.758+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:08.860 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:08.859+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:08.861 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:08.861 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:08.861 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:08.862 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:08.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:08.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '2 of 30' 2024-08-30T16:04:08.945 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:08.946 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:08.962 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:08.960+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:09.025 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:09.025 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:09.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:09.058 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:09.058 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:09.102 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:09.101+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:09.151 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:09.151 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:09.163 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:09.162+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:09.187 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:09.187 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:09.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:09.241 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:09.241 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:09.266 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:09.265+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:09.335 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:09.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:09.365 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:09.364+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:09.390 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:09.391 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:09.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:09.415 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:09.416 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:09.418 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:09.467 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:09.465+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:09.498 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:09.498 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:09.504 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:09.505 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:09.556 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:09.556 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:09.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:09.568 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:09.566+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:09.667 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:09.666+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:09.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-08-30T16:04:09.703 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:09.703 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:09.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:09.768 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:09.767+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:09.869 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:09.867+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:09.883 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:09.883 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:09.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:09.969 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:09.968+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:10.021 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:10.021 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:10.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:10.073 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:10.070+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:10.128 INFO:teuthology.orchestra.run.smithi050.stderr:set 2.7 pg_upmap mapping to [7,0] 2024-08-30T16:04:10.185 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:10.183+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:10.225 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:10.225 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:10.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:10.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '3 of 30' 2024-08-30T16:04:10.272 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:10.271+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:10.373 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:10.372+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:10.421 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:10.422 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:10.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:10.474 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:10.473+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:10.557 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:10.558 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:10.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:10.575 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:10.573+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:10.676 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:10.674+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:10.715 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:10.715 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:10.725 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:10.776 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:10.775+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:10.849 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:10.849 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:10.859 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:10.877 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:10.876+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:10.978 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:10.977+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:10.986 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:10.986 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:10.996 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:11.079 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:11.078+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:11.161 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:11.161 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:11.171 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:11.180 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:11.179+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:11.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '4 of 30' 2024-08-30T16:04:11.311 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:11.310+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:11.329 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:11.329 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:11.336 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-08-30T16:04:11.340 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:11.391 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:11.390+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:11.483 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:11.482+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:11.504 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:11.505 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:11.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:04:11.514 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:11.584 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:11.582+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:11.647 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:11.647 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:11.657 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:11.701 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:11.699+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:11.765 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:11.766 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:11.780 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:11.780 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:11.790 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:11.792 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:11.790+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:11.868 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:11.868 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:11.886 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:11.884+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:11.952 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:11.952 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:11.961 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:11.998 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:11.998 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:12.004 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:12.001+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:12.088 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:12.086+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:12.094 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:12.095 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:12.127 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:12.128 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:12.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:12.179 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:12.179 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:12.188 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:12.187+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:12.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '5 of 30' 2024-08-30T16:04:12.269 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:12.269 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:12.269 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:12.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:12.290 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:12.290 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:12.291 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:12.288+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:12.391 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:12.389+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:12.420 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:12.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:12.421 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:12.421 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:12.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:12.456 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:12.492 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:12.490+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:12.556 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:12.556 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:12.561 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:12.561 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:12.570 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:12.570 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:12.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:12.570 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:12.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:12.570 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(2.3 83/82 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 25'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@25'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:12.571 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:12.571 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.6612198710000001, 2024-08-30T16:04:12.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.10442118, 2024-08-30T16:04:12.571 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:12.571 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:12.571 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:12.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:12.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:12.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:12.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:12.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:12.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:12.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:12.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:12.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:12.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:12.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:12.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:12.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895800+0000", 2024-08-30T16:04:12.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999809 2024-08-30T16:04:12.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:12.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:12.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:12.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.921337+0000", 2024-08-30T16:04:12.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.02553677 2024-08-30T16:04:12.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:12.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:12.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:12.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.926424+0000", 2024-08-30T16:04:12.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050871340000000001 2024-08-30T16:04:12.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:12.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:12.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:12.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.935234+0000", 2024-08-30T16:04:12.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088098540000000006 2024-08-30T16:04:12.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:12.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:12.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:12.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.944321+0000", 2024-08-30T16:04:12.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0090873329999999995 2024-08-30T16:04:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.949613+0000", 2024-08-30T16:04:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052919710000000003 2024-08-30T16:04:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:08.000240+0000", 2024-08-30T16:04:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.050627411999999997 2024-08-30T16:04:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:12.574 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:12.575 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:12.575 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:12.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:12.620 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:12.619+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:12.693 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:12.692+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:12.751 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:12.751 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:12.752 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:12.752 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:12.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:12.801 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:12.800+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:12.895 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:12.894+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:12.928 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:12.928 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:12.928 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:12.929 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:12.929 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:12.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:13.021 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:13.020+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:13.096 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:13.095+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:13.112 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:13.112 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:13.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:13.112 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:13.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.113 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:13.113 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:13.113 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 291.83510503999997, 2024-08-30T16:04:13.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:04:13.113 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:13.113 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:13.114 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:13.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:13.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:13.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:13.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:13.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:13.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:13.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:13.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:04:13.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:04:13.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:04:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:04:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:04:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:04:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:04:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:04:13.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:13.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:04:13.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:04:13.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:13.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:04:13.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:04:13.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:13.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:04:13.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:04:13.118 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:13.118 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:13.118 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:13.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.119 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:13.119 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:13.119 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 291.562177679, 2024-08-30T16:04:13.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:04:13.119 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:13.119 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:13.119 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:13.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:13.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:13.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:13.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:13.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:13.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:13.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:13.122 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:04:13.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:04:13.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.122 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:13.122 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:04:13.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:04:13.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:13.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:04:13.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:04:13.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:13.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:04:13.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:04:13.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:13.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:04:13.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:04:13.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:13.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:04:13.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:04:13.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:13.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:04:13.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:04:13.127 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:13.127 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:13.127 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:13.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.127 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPull(2.3 e83/82 cost 1049576)", 2024-08-30T16:04:13.127 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:13.127 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.2780914409999999, 2024-08-30T16:04:13.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051106587000000002, 2024-08-30T16:04:13.127 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:13.127 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:13.127 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:13.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:13.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:13.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:13.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:13.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:13.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:13.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:13.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825962+0000", 2024-08-30T16:04:13.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:13.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:13.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.833441+0000", 2024-08-30T16:04:13.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0074797580000000004 2024-08-30T16:04:13.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:13.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.833651+0000", 2024-08-30T16:04:13.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000209414 2024-08-30T16:04:13.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:13.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.834053+0000", 2024-08-30T16:04:13.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040233199999999998 2024-08-30T16:04:13.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:13.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.835125+0000", 2024-08-30T16:04:13.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010716930000000001 2024-08-30T16:04:13.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:13.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.877088+0000", 2024-08-30T16:04:13.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.041962869999999999 2024-08-30T16:04:13.131 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:13.131 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:13.131 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:13.131 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:13.131 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:13.131 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:13.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:13.216 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:13.215+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:13.274 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:35.879 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:13.276 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:35.881 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:13.298 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:13.297+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:13.302 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:13.303 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:13.303 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:13.303 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:13.305 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:35.910 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:13.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:13.329 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:34.866 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:13.334 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:34.871 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:13.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-08-30T16:04:13.358 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:34.895 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:13.399 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:13.398+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:13.486 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:13.486 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:13.486 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:13.486 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:13.486 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:13.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:13.500 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:13.498+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:13.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '6 of 30' 2024-08-30T16:04:13.601 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:13.600+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:13.616 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:36.221 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:13.617 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:36.223 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:13.629 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:36.235 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:13.633 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:13.633 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:13.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:13.633 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:13.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.634 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:04:13.634 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:13.634 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 288.25594548999999, 2024-08-30T16:04:13.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:04:13.634 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:13.634 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:13.634 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:13.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:13.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:13.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:13.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:13.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:13.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:13.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:13.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:04:13.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:04:13.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:13.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:04:13.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:04:13.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:13.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:04:13.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:04:13.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:13.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:04:13.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:04:13.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:13.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:04:13.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:04:13.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:13.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:04:13.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:04:13.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:13.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:04:13.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:04:13.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:13.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:13.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:13.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:04:13.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:04:13.638 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:13.638 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:13.638 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:13.638 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:13.638 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:13.638 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:13.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:13.702 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:13.700+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:13.803 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:13.801+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:13.816 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:13.816 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:13.816 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:13.816 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:13.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:13.904 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:13.902+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:13.910 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:35.447 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:13.912 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:35.448 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:13.928 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:35.465 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:14.002 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:14.002 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:14.002 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:14.002 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:14.002 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:14.005 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:14.003+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:14.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:14.106 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:14.104+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:14.171 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:14.171 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:14.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:14.171 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:14.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.172 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:14.172 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:14.172 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 289.717517434, 2024-08-30T16:04:14.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:04:14.172 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:14.173 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:14.173 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:14.173 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:14.173 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:14.173 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:14.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.174 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:14.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:14.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:14.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:14.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:14.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:14.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:04:14.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:04:14.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:14.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:04:14.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:04:14.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:14.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:04:14.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:04:14.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:14.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:04:14.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:04:14.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:14.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:04:14.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:04:14.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:04:14.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:04:14.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:04:14.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:14.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:04:14.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:04:14.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:14.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:04:14.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:04:14.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:04:14.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:04:14.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:04:14.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:14.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:04:14.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:04:14.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:14.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:04:14.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:04:14.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:14.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:04:14.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:04:14.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:14.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:04:14.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:04:14.187 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.187 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:14.187 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.188 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:04:14.188 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:14.188 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 288.60165109899998, 2024-08-30T16:04:14.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:04:14.188 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:14.189 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:14.189 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:14.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:14.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:14.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:14.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:14.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:14.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:04:14.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:14.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:14.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:04:14.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:04:14.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:14.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:04:14.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:04:14.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:14.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:04:14.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:04:14.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:14.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:04:14.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:04:14.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:14.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:04:14.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:04:14.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:14.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:04:14.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:04:14.197 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.197 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:14.198 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.198 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:14.198 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:14.199 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 288.54205960100001, 2024-08-30T16:04:14.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:04:14.199 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:14.199 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:14.199 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:14.199 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:14.200 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:14.200 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:04:14.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.201 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:14.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.201 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:14.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:14.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:14.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:14.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:14.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:04:14.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:14.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:14.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:04:14.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:04:14.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.205 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:14.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:04:14.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:04:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:04:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:04:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:04:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:04:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:14.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:04:14.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:04:14.207 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.207 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:14.207 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.207 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:04:14.207 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:14.207 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 288.51536091399998, 2024-08-30T16:04:14.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:04:14.207 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:14.207 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:14.208 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:14.208 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:14.208 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:14.208 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:14.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.208 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:14.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:14.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:14.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:14.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:14.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:14.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:04:14.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:14.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:14.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:04:14.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:04:14.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:14.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:04:14.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:04:14.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:14.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:04:14.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:04:14.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:14.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:04:14.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:04:14.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:14.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:04:14.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:04:14.211 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.211 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:14.211 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.212 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:14.212 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:14.212 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 287.647744386, 2024-08-30T16:04:14.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:04:14.212 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:14.212 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:14.212 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:14.212 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:14.212 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:14.212 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:14.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.212 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:14.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:14.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:14.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:14.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:14.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:14.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:14.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:04:14.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:04:14.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:14.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:04:14.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:04:14.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:14.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:04:14.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:04:14.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:14.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:04:14.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:04:14.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:14.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:04:14.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:04:14.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:14.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.218 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:04:14.218 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:14.218 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 287.48849730699999, 2024-08-30T16:04:14.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:04:14.218 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:14.218 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:14.218 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:14.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:14.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:14.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:14.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:14.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:14.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:04:14.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:04:14.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:14.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:04:14.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:04:14.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:14.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:04:14.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:04:14.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:14.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:04:14.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:04:14.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:14.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:04:14.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:04:14.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:14.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:04:14.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:04:14.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.222 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:14.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:04:14.222 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:04:14.222 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.222 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:14.222 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.222 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:14.222 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:14.222 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 287.21232324099998, 2024-08-30T16:04:14.222 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:04:14.223 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:14.223 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:14.223 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:14.223 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:14.223 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:14.223 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:14.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.223 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:14.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.223 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:14.223 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:14.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:14.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:14.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:14.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:04:14.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:14.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:14.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:04:14.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:04:14.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:14.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:04:14.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:04:14.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:14.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:04:14.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:04:14.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:14.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:04:14.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:04:14.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:14.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:04:14.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:04:14.226 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.226 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:14.226 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.227 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:14.227 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:14.227 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 286.554031662, 2024-08-30T16:04:14.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:04:14.227 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:14.227 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:14.227 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:14.227 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:14.227 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:14.227 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:04:14.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.227 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:14.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:14.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:14.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:14.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:14.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:14.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:04:14.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:14.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:14.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:04:14.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:04:14.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:14.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:04:14.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:04:14.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:14.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:04:14.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:04:14.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:14.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:04:14.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:04:14.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:14.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:04:14.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:04:14.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:14.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:04:14.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:04:14.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:14.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:04:14.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:04:14.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:14.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:04:14.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:04:14.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:14.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:04:14.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:04:14.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:14.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:04:14.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:04:14.232 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.232 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:14.232 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.233 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:04:14.233 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:14.233 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 286.45811837799999, 2024-08-30T16:04:14.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:04:14.233 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:14.233 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:14.233 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:14.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:14.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:14.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:14.234 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:14.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:14.234 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:04:14.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:14.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:04:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:04:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:04:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:04:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:04:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:04:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:14.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:04:14.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:04:14.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:14.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:04:14.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:04:14.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:14.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:04:14.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:04:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 227.20675786699999, 2024-08-30T16:04:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:04:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:14.238 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:14.238 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:14.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.238 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:14.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:14.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:14.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:14.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:14.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:14.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:04:14.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:04:14.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:14.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:04:14.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:04:14.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:14.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:04:14.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:04:14.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:14.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:04:14.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:04:14.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:14.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:04:14.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:04:14.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:14.241 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:04:14.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:04:14.241 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.241 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:14.241 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.241 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:04:14.241 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:14.241 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 167.19765318, 2024-08-30T16:04:14.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:04:14.241 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:04:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:04:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:04:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:04:14.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:14.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:04:14.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:04:14.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:14.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:04:14.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:04:14.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:14.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:04:14.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:04:14.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:14.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:14.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:14.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:04:14.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:04:14.245 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.245 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:14.245 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.245 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:14.245 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:14.245 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:14.247 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:14.247 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:14.245+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:14.348 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:14.347+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:14.375 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:14.375 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:14.375 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:14.375 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:14.384 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:14.449 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:14.448+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:14.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '7 of 30' 2024-08-30T16:04:14.550 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:14.550 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:14.550 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:14.551 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:14.551 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:14.553 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:14.552+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:14.559 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:14.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:04:14.591 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-08-30T16:04:14.651 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:14.650+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:14.690 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:14.690 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:14.690 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:14.690 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:04:14.690 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:14.699 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:14.752 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:14.750+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:14.807 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:14.807 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:14.826 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:14.827 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:14.827 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:14.836 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:14.854 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:14.852+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:14.900 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:14.900 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:14.954 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:14.953+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:14.988 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:14.988 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:15.000 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:15.000 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:15.000 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:15.009 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:15.056 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:15.054+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:15.077 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:15.078 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:15.147 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7] out_osds: [6, 4, 2] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-08-30T16:04:15.147 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:04:15.148 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-08-30T16:04:15.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 0.05 2024-08-30T16:04:15.172 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:15.172 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:15.193 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:15.193 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:15.193 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:15.193 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:15.193 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.193 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:15.193 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:15.194 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 304.33022292099997, 2024-08-30T16:04:15.194 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:04:15.194 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.194 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.194 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.194 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.194 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.194 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:04:15.194 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.194 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.194 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.194 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.194 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:15.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:15.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.195 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.195 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:04:15.195 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:04:15.195 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.195 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:04:15.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:04:15.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:04:15.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:04:15.196 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.196 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.196 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.196 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:04:15.196 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:04:15.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:04:15.197 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:04:15.197 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.197 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.197 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:04:15.197 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:04:15.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:04:15.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:04:15.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:04:15.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.198 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.198 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:04:15.198 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:04:15.198 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.198 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:04:15.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:04:15.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:04:15.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:04:15.199 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.199 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.199 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.199 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:04:15.199 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:04:15.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.200 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.200 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.200 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.200 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.200 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:15.200 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:15.200 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 303.41380690099999, 2024-08-30T16:04:15.200 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:04:15.200 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.200 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.200 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.200 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.201 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.201 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:04:15.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.201 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.201 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.201 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.201 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:15.201 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.201 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.202 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:15.202 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.202 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.202 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.202 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:04:15.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:15.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:04:15.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:04:15.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.203 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.203 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.203 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:04:15.203 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:04:15.203 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:04:15.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:04:15.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:04:15.204 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:04:15.204 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.204 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.204 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.204 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:04:15.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:04:15.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:04:15.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:04:15.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.205 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:04:15.205 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:04:15.205 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.205 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.205 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.206 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:04:15.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:04:15.206 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.206 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.206 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.206 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.206 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.206 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:15.206 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:15.206 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 303.41210234800002, 2024-08-30T16:04:15.206 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:04:15.206 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.206 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.207 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.207 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.207 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.207 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:04:15.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.207 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.207 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:15.207 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.207 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.207 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.207 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:15.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:04:15.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:04:15.208 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.208 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.208 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.208 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:04:15.208 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:04:15.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:04:15.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:04:15.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.209 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:04:15.209 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:04:15.209 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.209 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.209 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:04:15.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:04:15.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:04:15.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:04:15.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.210 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.210 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.210 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:04:15.210 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:04:15.210 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.211 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:04:15.211 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:04:15.211 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.211 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.211 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.212 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:04:15.212 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:04:15.212 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.212 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.212 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.212 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.212 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.212 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.212 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:15.213 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 302.99264854099999, 2024-08-30T16:04:15.213 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:04:15.213 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.213 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.213 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.213 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.213 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.213 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:04:15.213 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.214 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:15.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.214 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:15.214 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.214 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.214 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.214 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:04:15.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:04:15.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:04:15.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:04:15.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.215 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.215 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.215 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:04:15.215 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:04:15.215 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:04:15.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:04:15.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:04:15.216 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:04:15.216 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.216 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.216 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.216 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:04:15.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:04:15.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:04:15.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:04:15.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.217 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.217 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:04:15.217 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:04:15.217 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.217 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.218 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.218 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:04:15.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:04:15.218 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.218 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.218 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.218 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.218 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.218 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.218 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:15.218 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 302.97576036700002, 2024-08-30T16:04:15.218 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:04:15.218 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.219 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.219 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.219 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.219 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.219 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:04:15.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.219 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.219 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.219 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:15.219 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.219 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.219 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:15.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:04:15.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:04:15.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.220 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.220 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.220 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:04:15.220 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:04:15.220 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:04:15.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:04:15.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:04:15.221 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:04:15.221 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.221 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.221 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:04:15.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:04:15.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:04:15.222 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:04:15.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.222 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.222 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:04:15.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:04:15.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:04:15.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:04:15.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.223 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.223 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:04:15.223 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:04:15.223 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.223 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.223 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.224 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.224 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:15.224 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 302.96276029799998, 2024-08-30T16:04:15.224 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:04:15.224 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.224 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.224 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.224 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.224 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.225 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:04:15.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.225 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.225 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:15.225 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.225 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:15.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:04:15.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:04:15.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.226 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.226 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:04:15.226 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:04:15.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:04:15.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:04:15.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.227 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:04:15.227 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:04:15.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.227 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:04:15.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:04:15.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:04:15.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:04:15.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.228 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.228 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:04:15.228 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:04:15.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:04:15.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:04:15.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.229 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.229 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:04:15.229 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:04:15.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.229 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.230 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.230 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:15.230 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 302.96026800599998, 2024-08-30T16:04:15.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:04:15.230 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.230 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.230 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.230 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:04:15.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.230 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:15.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:15.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:04:15.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:15.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:04:15.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:04:15.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:04:15.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:04:15.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:04:15.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:04:15.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:04:15.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:04:15.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:04:15.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:04:15.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:04:15.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:04:15.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:04:15.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:04:15.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.235 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.235 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:04:15.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:04:15.235 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.235 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.235 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.235 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.235 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:15.235 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 302.95951055799998, 2024-08-30T16:04:15.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:04:15.236 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.236 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.236 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.236 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.236 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.236 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:04:15.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.236 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:15.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:15.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:04:15.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:04:15.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:04:15.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:04:15.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:04:15.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:04:15.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:04:15.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:04:15.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:04:15.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:04:15.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.239 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:04:15.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:04:15.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.239 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:04:15.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:04:15.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:04:15.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:04:15.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:04:15.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:04:15.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.241 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.241 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.241 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:15.241 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 302.95877810799999, 2024-08-30T16:04:15.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:04:15.241 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.242 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.242 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.242 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.242 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.242 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:04:15.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.242 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:15.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:15.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:04:15.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:15.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:04:15.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:04:15.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:04:15.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:04:15.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:04:15.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:04:15.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:04:15.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:04:15.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:04:15.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:04:15.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:04:15.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:04:15.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:04:15.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:04:15.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:04:15.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:04:15.246 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.246 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.246 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.247 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.247 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:15.247 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 302.95623930900001, 2024-08-30T16:04:15.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:04:15.247 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.247 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.247 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.247 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.248 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.248 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:04:15.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.248 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:15.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:15.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:04:15.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:15.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:04:15.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:04:15.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:04:15.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:04:15.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:04:15.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:04:15.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:04:15.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:04:15.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:04:15.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:04:15.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:04:15.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:04:15.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:04:15.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:04:15.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:04:15.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:04:15.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.252 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.252 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.252 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.252 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:15.252 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 302.95549130799998, 2024-08-30T16:04:15.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:04:15.253 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.253 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.253 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.253 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.254 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.254 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:04:15.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.254 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:15.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:15.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:04:15.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:04:15.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:04:15.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:04:15.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:04:15.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:04:15.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:04:15.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:04:15.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:04:15.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:04:15.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:04:15.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:04:15.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:04:15.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:04:15.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:04:15.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:04:15.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:04:15.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:04:15.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.258 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.258 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.259 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.259 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:15.259 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 302.18337483599998, 2024-08-30T16:04:15.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:04:15.259 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.259 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.259 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.259 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:04:15.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.259 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:15.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:15.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.260 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.260 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.260 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:04:15.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:15.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:04:15.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:04:15.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:04:15.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:04:15.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:04:15.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:04:15.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:04:15.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:04:15.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:04:15.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:04:15.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:04:15.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:04:15.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:04:15.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:04:15.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:04:15.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:04:15.264 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.265 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.265 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.265 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.265 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:15.265 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 302.18131769000001, 2024-08-30T16:04:15.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:04:15.265 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.265 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.265 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.265 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.265 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.266 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:04:15.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.266 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:15.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:15.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:04:15.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:15.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:04:15.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:04:15.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:04:15.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:04:15.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:04:15.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:04:15.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:04:15.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:04:15.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:04:15.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:04:15.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:04:15.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:04:15.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:04:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:04:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:04:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:04:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.270 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:15.271 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 301.91837319000001, 2024-08-30T16:04:15.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:04:15.271 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.271 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.271 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.271 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.271 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.271 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:04:15.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.271 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:15.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:04:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:04:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:04:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:04:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:04:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:04:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:04:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:04:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:04:15.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:04:15.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:04:15.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:04:15.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:04:15.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:04:15.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:04:15.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:04:15.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:04:15.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:04:15.275 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.275 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.275 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.276 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.276 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:15.276 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 301.91728168700001, 2024-08-30T16:04:15.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:04:15.276 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.276 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.276 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.276 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.276 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.276 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:04:15.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.276 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:15.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:15.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:04:15.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:15.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:04:15.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:04:15.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:04:15.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:04:15.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.278 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:04:15.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:04:15.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:04:15.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:04:15.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:04:15.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:04:15.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:04:15.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:04:15.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:04:15.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:04:15.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:04:15.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:04:15.280 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.281 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.281 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.281 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.281 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:15.281 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 301.91653210200002, 2024-08-30T16:04:15.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:04:15.281 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.281 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.281 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.281 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:04:15.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.282 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:15.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:15.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:04:15.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:15.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:04:15.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:04:15.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:04:15.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:04:15.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:04:15.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:04:15.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:04:15.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:04:15.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:04:15.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:04:15.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:04:15.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:04:15.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:04:15.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:04:15.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:04:15.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:04:15.286 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.286 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.286 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.286 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.286 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:15.286 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 301.88280056999997, 2024-08-30T16:04:15.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:04:15.286 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.286 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.286 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.287 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.287 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.287 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:04:15.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.287 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:15.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.287 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.287 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:15.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:04:15.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:04:15.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:04:15.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:04:15.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:04:15.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:04:15.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:04:15.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:04:15.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:04:15.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:04:15.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:04:15.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:04:15.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:04:15.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:04:15.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:04:15.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:04:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:04:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:04:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 301.88227456200002, 2024-08-30T16:04:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:04:15.292 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.292 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.292 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.292 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.292 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.292 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:04:15.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.292 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.292 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.292 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.292 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:15.292 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.292 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:15.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:04:15.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:15.293 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.293 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.293 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.293 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:04:15.293 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:04:15.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:04:15.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:04:15.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:04:15.294 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:04:15.294 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.294 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.294 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.294 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:04:15.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:04:15.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:04:15.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:04:15.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.295 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:04:15.295 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:04:15.295 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.295 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.295 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:04:15.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:04:15.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.296 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.296 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:04:15.296 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:04:15.296 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.296 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.296 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.296 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.296 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.296 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:15.297 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:15.297 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 301.88181899900002, 2024-08-30T16:04:15.297 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:04:15.297 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.297 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.297 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.297 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.297 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.297 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:04:15.297 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.297 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.297 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.297 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:15.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:15.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.298 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.298 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.298 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:04:15.298 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:04:15.298 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:04:15.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:04:15.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:04:15.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:04:15.299 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.299 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.299 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.299 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:04:15.299 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:04:15.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:04:15.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:04:15.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.300 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:04:15.300 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:04:15.300 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.300 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.300 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:04:15.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:04:15.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.301 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:04:15.301 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:04:15.301 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.301 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.301 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.302 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:04:15.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:04:15.302 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.302 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.302 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.302 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.302 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.302 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:04:15.302 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:15.302 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 297.21130368299998, 2024-08-30T16:04:15.302 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:04:15.302 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:15.303 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:15.303 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:15.303 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:15.303 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:15.303 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:04:15.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.303 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:15.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:15.303 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:15.303 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.303 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.303 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.303 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:15.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:15.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:15.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:15.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:04:15.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:15.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.304 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.304 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:15.304 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:04:15.304 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:04:15.304 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:15.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:04:15.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:04:15.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:15.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:04:15.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:04:15.305 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.305 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.305 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:15.305 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:04:15.305 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:04:15.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:15.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:04:15.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:04:15.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:15.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:04:15.306 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:04:15.306 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.306 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.306 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:15.306 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:04:15.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:04:15.307 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:15.307 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:15.307 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:15.307 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:04:15.307 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:04:15.307 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.307 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.307 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.307 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:15.307 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:15.307 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:15.308 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:15.191+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:15.309 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:15.312 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:15.350 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:15.348+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:15.414 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:15.413+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:15.490 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:15.491 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:15.491 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:15.491 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:15.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:15.516 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:15.514+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:15.524 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:15.614 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:15.615 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:15.616 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:15.614+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:15.620 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:15.620 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:15.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '8 of 30' 2024-08-30T16:04:15.670 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:15.670 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:15.680 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:15.721 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:15.719+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:15.734 INFO:teuthology.orchestra.run.smithi050.stdout:moved 0 / 18 (0%) 2024-08-30T16:04:15.734 INFO:teuthology.orchestra.run.smithi050.stdout:avg 3.6 2024-08-30T16:04:15.734 INFO:teuthology.orchestra.run.smithi050.stdout:stddev 1.0198 -> 1.0198 (expected baseline 1.69706) 2024-08-30T16:04:15.734 INFO:teuthology.orchestra.run.smithi050.stdout:min osd.1 with 2 -> 2 pgs (0.555556 -> 0.555556 * mean) 2024-08-30T16:04:15.734 INFO:teuthology.orchestra.run.smithi050.stdout:max osd.5 with 5 -> 5 pgs (1.38889 -> 1.38889 * mean) 2024-08-30T16:04:15.734 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:04:15.734 INFO:teuthology.orchestra.run.smithi050.stdout:oload 110 2024-08-30T16:04:15.734 INFO:teuthology.orchestra.run.smithi050.stdout:max_change 0.05 2024-08-30T16:04:15.734 INFO:teuthology.orchestra.run.smithi050.stdout:max_change_osds 4 2024-08-30T16:04:15.735 INFO:teuthology.orchestra.run.smithi050.stdout:average_utilization 0.0003 2024-08-30T16:04:15.735 INFO:teuthology.orchestra.run.smithi050.stdout:overload_utilization 0.0003 2024-08-30T16:04:15.735 INFO:teuthology.orchestra.run.smithi050.stderr:no change 2024-08-30T16:04:15.748 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-08-30T16:04:15.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2024-08-30T16:04:15.818 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:15.817+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:15.849 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:15.849 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:15.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:15.919 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:15.917+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:15.986 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:15.986 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:15.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:16.020 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:16.018+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:16.121 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:16.120+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:16.125 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:16.126 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:16.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:16.222 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:16.221+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:16.303 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:16.303 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:16.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:16.313 INFO:teuthology.orchestra.run.smithi050.stdout:moved 0 / 18 (0%) 2024-08-30T16:04:16.313 INFO:teuthology.orchestra.run.smithi050.stdout:avg 3.6 2024-08-30T16:04:16.313 INFO:teuthology.orchestra.run.smithi050.stdout:stddev 1.0198 -> 1.0198 (expected baseline 1.69706) 2024-08-30T16:04:16.314 INFO:teuthology.orchestra.run.smithi050.stdout:min osd.1 with 2 -> 2 pgs (0.555556 -> 0.555556 * mean) 2024-08-30T16:04:16.314 INFO:teuthology.orchestra.run.smithi050.stdout:max osd.5 with 5 -> 5 pgs (1.38889 -> 1.38889 * mean) 2024-08-30T16:04:16.314 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:04:16.314 INFO:teuthology.orchestra.run.smithi050.stdout:oload 1000 2024-08-30T16:04:16.314 INFO:teuthology.orchestra.run.smithi050.stdout:max_change 0.05 2024-08-30T16:04:16.314 INFO:teuthology.orchestra.run.smithi050.stdout:max_change_osds 4 2024-08-30T16:04:16.314 INFO:teuthology.orchestra.run.smithi050.stdout:average_utilization 0.0003 2024-08-30T16:04:16.314 INFO:teuthology.orchestra.run.smithi050.stdout:overload_utilization 0.0029 2024-08-30T16:04:16.314 INFO:teuthology.orchestra.run.smithi050.stderr:no change 2024-08-30T16:04:16.323 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:16.321+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:16.326 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-08-30T16:04:16.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 3.0 2024-08-30T16:04:16.424 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:16.422+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:16.444 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:16.444 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:16.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:16.524 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:16.523+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:16.583 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:16.583 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:16.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:16.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-08-30T16:04:16.636 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:16.635+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:16.724 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:16.724 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:16.727 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:16.725+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:16.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:16.828 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:16.826+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:16.885 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:16.886 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:16.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:16.939 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:16.937+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:16.970 INFO:teuthology.orchestra.run.smithi050.stdout:moved 0 / 18 (0%) 2024-08-30T16:04:16.970 INFO:teuthology.orchestra.run.smithi050.stdout:avg 3.6 2024-08-30T16:04:16.970 INFO:teuthology.orchestra.run.smithi050.stdout:stddev 1.0198 -> 1.0198 (expected baseline 1.69706) 2024-08-30T16:04:16.970 INFO:teuthology.orchestra.run.smithi050.stdout:min osd.1 with 2 -> 2 pgs (0.555556 -> 0.555556 * mean) 2024-08-30T16:04:16.970 INFO:teuthology.orchestra.run.smithi050.stdout:max osd.5 with 5 -> 5 pgs (1.38889 -> 1.38889 * mean) 2024-08-30T16:04:16.970 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:04:16.971 INFO:teuthology.orchestra.run.smithi050.stdout:oload 1000 2024-08-30T16:04:16.971 INFO:teuthology.orchestra.run.smithi050.stdout:max_change 3 2024-08-30T16:04:16.971 INFO:teuthology.orchestra.run.smithi050.stdout:max_change_osds 4 2024-08-30T16:04:16.971 INFO:teuthology.orchestra.run.smithi050.stdout:average_utilization 0.0003 2024-08-30T16:04:16.971 INFO:teuthology.orchestra.run.smithi050.stdout:overload_utilization 0.0029 2024-08-30T16:04:16.971 INFO:teuthology.orchestra.run.smithi050.stderr:no change 2024-08-30T16:04:16.983 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-08-30T16:04:16.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 1.0 2024-08-30T16:04:17.029 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:17.027+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:17.037 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:17.037 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:17.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:17.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '9 of 30' 2024-08-30T16:04:17.131 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:17.129+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:17.231 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:17.229+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:17.262 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:17.262 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:17.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:17.356 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:17.355+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:17.432 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:17.430+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:17.439 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:17.439 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:17.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:17.533 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:17.532+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:17.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:04:17.632 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:17.632 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:17.633 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:17.632+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:17.636 INFO:teuthology.orchestra.run.smithi050.stdout:moved 0 / 18 (0%) 2024-08-30T16:04:17.636 INFO:teuthology.orchestra.run.smithi050.stdout:avg 3.6 2024-08-30T16:04:17.636 INFO:teuthology.orchestra.run.smithi050.stdout:stddev 1.0198 -> 1.0198 (expected baseline 1.69706) 2024-08-30T16:04:17.636 INFO:teuthology.orchestra.run.smithi050.stdout:min osd.1 with 2 -> 2 pgs (0.555556 -> 0.555556 * mean) 2024-08-30T16:04:17.637 INFO:teuthology.orchestra.run.smithi050.stdout:max osd.5 with 5 -> 5 pgs (1.38889 -> 1.38889 * mean) 2024-08-30T16:04:17.637 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:04:17.637 INFO:teuthology.orchestra.run.smithi050.stdout:oload 1000 2024-08-30T16:04:17.637 INFO:teuthology.orchestra.run.smithi050.stdout:max_change 1 2024-08-30T16:04:17.637 INFO:teuthology.orchestra.run.smithi050.stdout:max_change_osds 4 2024-08-30T16:04:17.637 INFO:teuthology.orchestra.run.smithi050.stdout:average_utilization 0.0003 2024-08-30T16:04:17.637 INFO:teuthology.orchestra.run.smithi050.stdout:overload_utilization 0.0029 2024-08-30T16:04:17.637 INFO:teuthology.orchestra.run.smithi050.stderr:no change 2024-08-30T16:04:17.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:17.652 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-08-30T16:04:17.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 1.0 2024-08-30T16:04:17.736 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:17.735+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:17.835 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:17.834+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:17.837 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:17.837 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:17.847 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:17.875 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:17.875 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:17.936 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:17.934+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:17.967 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:17.967 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:17.985 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:17.986 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:17.994 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:18.037 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:18.035+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:18.055 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:18.056 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:18.137 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:18.136+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:18.147 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:18.147 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:18.183 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:18.183 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:18.192 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:18.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '10 of 30' 2024-08-30T16:04:18.259 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:18.259 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:18.260 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:18.257+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:18.299 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-08-30T16:04:18.308 INFO:teuthology.orchestra.run.smithi050.stdout:moved 0 / 18 (0%) 2024-08-30T16:04:18.308 INFO:teuthology.orchestra.run.smithi050.stdout:avg 3.6 2024-08-30T16:04:18.309 INFO:teuthology.orchestra.run.smithi050.stdout:stddev 1.0198 -> 1.0198 (expected baseline 1.69706) 2024-08-30T16:04:18.309 INFO:teuthology.orchestra.run.smithi050.stdout:min osd.1 with 2 -> 2 pgs (0.555556 -> 0.555556 * mean) 2024-08-30T16:04:18.309 INFO:teuthology.orchestra.run.smithi050.stdout:max osd.5 with 5 -> 5 pgs (1.38889 -> 1.38889 * mean) 2024-08-30T16:04:18.309 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:04:18.309 INFO:teuthology.orchestra.run.smithi050.stdout:oload 110 2024-08-30T16:04:18.309 INFO:teuthology.orchestra.run.smithi050.stdout:max_change 1 2024-08-30T16:04:18.309 INFO:teuthology.orchestra.run.smithi050.stdout:max_change_osds 4 2024-08-30T16:04:18.309 INFO:teuthology.orchestra.run.smithi050.stdout:average_utilization 0.0003 2024-08-30T16:04:18.309 INFO:teuthology.orchestra.run.smithi050.stdout:overload_utilization 0.0003 2024-08-30T16:04:18.309 INFO:teuthology.orchestra.run.smithi050.stderr:no change 2024-08-30T16:04:18.331 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:18.332 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:04:18.332 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:18.339 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:18.338+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:18.340 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:18.354 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:18.354 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:18.409 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:18.440 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:18.439+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:18.507 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:18.508 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:18.511 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:18.522 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:18.522 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:18.522 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:18.522 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:18.531 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:18.576 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:18.574+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:18.643 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:18.641+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:18.653 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:18.654 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:18.654 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:18.654 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:18.654 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:18.663 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:18.744 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:18.742+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:18.845 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:18.844+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:18.904 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:18.905 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:18.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:18.905 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:18.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.905 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:18.905 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:18.905 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 308.04401003599997, 2024-08-30T16:04:18.905 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:04:18.905 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.905 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.905 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.905 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.906 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.906 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:04:18.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.906 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:18.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.906 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.906 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:18.906 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:04:18.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:04:18.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.907 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.907 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:04:18.907 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:04:18.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:04:18.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:04:18.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:04:18.908 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:04:18.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.908 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.908 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:04:18.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:04:18.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:04:18.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:04:18.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:04:18.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.909 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.909 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:04:18.909 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:04:18.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:04:18.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:04:18.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:04:18.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:04:18.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.910 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.910 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:04:18.910 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:04:18.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.911 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.911 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:04:18.911 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:04:18.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.911 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.911 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.911 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.911 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:18.911 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:18.911 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 307.12759401599999, 2024-08-30T16:04:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:04:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:04:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.912 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:18.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:18.913 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.913 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.913 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.913 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.913 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:04:18.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:18.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.914 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:04:18.914 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:04:18.914 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.914 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.914 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:04:18.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:04:18.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.915 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:04:18.915 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:04:18.915 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.915 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.915 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:04:18.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:04:18.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:04:18.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:04:18.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.916 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.916 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.916 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:04:18.916 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:04:18.916 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:04:18.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:04:18.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.917 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.917 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.917 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:04:18.917 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:04:18.917 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.917 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.917 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.917 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.918 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.918 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:18.918 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:18.918 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 307.12588946300002, 2024-08-30T16:04:18.918 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:04:18.918 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.918 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.918 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.918 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.918 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.918 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:04:18.918 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.918 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:18.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:18.919 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.919 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.919 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.919 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.919 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:04:18.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:04:18.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:04:18.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:04:18.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.920 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.920 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.920 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:04:18.920 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:04:18.920 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:04:18.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:04:18.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:04:18.921 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:04:18.921 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.921 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.921 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.921 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:04:18.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:04:18.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:04:18.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:04:18.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.922 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:04:18.922 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:04:18.922 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.922 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.922 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.923 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:04:18.923 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:04:18.923 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.923 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.923 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.923 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.923 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.923 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:18.923 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:18.924 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 306.706435656, 2024-08-30T16:04:18.924 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:04:18.924 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.924 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.925 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.925 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.925 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.925 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:04:18.925 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.925 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.925 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.925 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.925 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:18.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:18.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.926 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.926 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.926 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.926 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:04:18.926 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:04:18.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:04:18.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:04:18.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.927 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.927 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.927 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:04:18.927 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:04:18.927 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:04:18.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:04:18.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.928 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.928 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.928 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:04:18.928 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:04:18.928 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:04:18.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:04:18.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.929 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.929 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.929 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:04:18.929 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:04:18.929 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:04:18.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:04:18.930 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.930 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.930 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.930 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:04:18.930 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:04:18.930 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.930 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.931 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.931 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.931 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.931 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:18.931 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:18.931 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 306.68954748200002, 2024-08-30T16:04:18.931 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:04:18.931 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.931 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.931 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.931 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.931 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.931 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:04:18.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.932 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:18.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.932 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.932 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.932 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:18.932 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.932 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:04:18.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:04:18.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.933 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.933 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:04:18.933 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:04:18.933 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.933 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:04:18.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:04:18.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:04:18.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:04:18.934 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.934 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.934 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.934 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:04:18.934 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:04:18.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:04:18.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:04:18.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.935 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:04:18.935 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:04:18.935 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.935 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.935 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:04:18.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:04:18.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.936 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.936 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:04:18.936 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:04:18.936 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.936 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.936 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.936 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.936 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.936 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:18.937 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:18.937 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 306.67654741299998, 2024-08-30T16:04:18.937 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:04:18.937 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.937 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.937 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.937 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.937 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.937 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:04:18.937 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.937 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.937 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.937 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:18.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:18.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.938 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.938 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.938 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.938 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:04:18.938 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:04:18.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:04:18.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:04:18.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:04:18.939 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:04:18.939 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.939 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.939 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.939 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:04:18.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:04:18.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:04:18.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:04:18.940 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.940 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.940 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.940 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:04:18.940 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:04:18.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:04:18.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:04:18.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.941 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.941 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:04:18.941 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:04:18.941 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.941 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.942 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.942 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:04:18.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:04:18.942 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.942 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.942 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.942 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.942 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.942 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:18.942 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:18.942 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 306.67405512099998, 2024-08-30T16:04:18.942 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:04:18.943 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.943 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.943 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.943 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.943 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.943 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:04:18.943 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.944 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:18.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.944 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.944 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:18.944 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.944 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.944 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:04:18.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:18.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:04:18.945 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:04:18.945 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.945 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.945 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.945 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:04:18.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:04:18.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:04:18.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:04:18.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.946 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:04:18.946 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:04:18.946 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.946 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.946 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:04:18.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:04:18.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:04:18.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:04:18.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.947 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.947 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.947 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:04:18.947 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:04:18.947 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.948 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.948 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:04:18.948 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:04:18.948 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.948 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.948 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.948 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.948 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.948 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:18.948 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:18.948 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 306.67329767299998, 2024-08-30T16:04:18.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:04:18.949 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.949 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.949 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.949 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.949 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.949 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:04:18.949 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.949 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.949 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.949 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.949 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:18.949 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:18.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:04:18.950 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:04:18.950 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.950 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.950 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.950 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.951 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:04:18.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:04:18.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:04:18.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:04:18.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.952 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.952 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:04:18.952 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:04:18.952 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.952 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:04:18.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:04:18.953 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.953 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.953 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.953 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:04:18.953 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:04:18.954 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.954 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.954 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.954 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.955 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:18.955 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:18.955 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 306.67256522299999, 2024-08-30T16:04:18.955 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:04:18.955 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.955 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.955 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.955 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.955 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.955 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:04:18.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.955 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:18.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.956 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:18.956 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.956 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:04:18.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:18.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:04:18.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:04:18.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.957 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.957 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:04:18.957 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:04:18.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:04:18.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:04:18.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.958 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:04:18.958 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:04:18.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.958 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:04:18.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:04:18.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:04:18.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:04:18.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.959 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.959 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:04:18.959 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:04:18.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.960 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.960 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:04:18.960 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:04:18.960 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.960 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.960 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.960 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:18.960 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:18.960 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 306.67002642400001, 2024-08-30T16:04:18.961 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:04:18.961 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.961 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.961 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.961 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.961 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.961 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:04:18.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.961 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.961 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.961 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:18.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:18.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.962 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.962 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:04:18.962 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:18.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:04:18.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:04:18.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:04:18.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:04:18.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.963 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.963 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:04:18.963 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:04:18.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:04:18.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:04:18.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.964 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.964 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:04:18.964 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:04:18.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:04:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:04:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:04:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:04:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:04:18.965 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:04:18.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.966 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.966 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.966 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.966 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:18.966 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:18.966 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 306.66927842299998, 2024-08-30T16:04:18.966 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:04:18.966 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.966 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.966 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.966 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.967 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.967 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:04:18.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.967 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:18.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.967 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.967 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.967 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.967 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:18.967 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:04:18.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:04:18.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.968 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.968 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:04:18.968 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:04:18.968 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:04:18.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:04:18.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:04:18.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:04:18.969 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.969 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.969 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.969 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:04:18.969 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:04:18.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:04:18.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:04:18.970 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.970 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.970 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.970 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:04:18.970 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:04:18.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:04:18.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:04:18.971 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.971 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.971 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.971 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:04:18.971 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:04:18.971 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.971 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.971 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.972 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.972 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.972 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:18.972 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:18.972 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 305.89716195099999, 2024-08-30T16:04:18.972 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:04:18.972 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.972 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.972 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.972 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:04:18.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.973 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:18.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.973 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.973 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:18.973 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.973 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.973 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:04:18.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:18.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:04:18.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:04:18.974 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.974 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.974 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.974 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:04:18.974 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:04:18.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:04:18.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:04:18.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.975 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:04:18.975 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:04:18.975 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.975 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.975 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:04:18.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:04:18.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:04:18.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:04:18.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.976 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.976 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.976 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:04:18.976 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:04:18.976 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.977 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.977 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:04:18.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:04:18.977 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.977 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.977 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.977 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.977 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.977 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:18.977 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:18.977 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 305.89510480500002, 2024-08-30T16:04:18.977 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:04:18.978 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.978 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.978 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.978 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.978 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.978 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:04:18.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.978 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.978 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.978 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.978 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:18.978 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.978 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:18.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:04:18.979 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:18.979 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.979 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.979 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.979 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:04:18.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:04:18.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:04:18.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:04:18.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.980 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.980 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.980 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:04:18.980 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:04:18.980 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:04:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:04:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:04:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:04:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:04:18.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:04:18.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:04:18.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:04:18.982 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.982 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.982 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.982 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:04:18.982 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:04:18.982 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.982 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.983 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.983 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.983 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.983 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:18.983 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:18.983 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 305.63216030500001, 2024-08-30T16:04:18.983 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:04:18.983 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.983 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.983 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.983 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:04:18.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.984 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:18.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.984 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.984 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:18.984 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.984 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.984 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:04:18.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:04:18.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:04:18.985 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:04:18.985 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.985 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.985 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.985 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:04:18.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:04:18.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:04:18.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:04:18.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.986 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.986 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.986 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:04:18.986 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:04:18.986 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:04:18.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:04:18.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.987 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:04:18.987 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:04:18.987 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.987 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.987 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:04:18.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:04:18.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.988 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.988 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:04:18.988 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:04:18.988 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.988 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.988 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.988 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.988 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.988 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:18.989 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:18.989 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 305.63106880200002, 2024-08-30T16:04:18.989 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:04:18.989 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.989 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.989 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.989 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.989 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.989 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:04:18.989 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.989 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.989 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:18.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:18.990 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.990 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.990 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.990 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.990 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:04:18.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:18.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:04:18.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:04:18.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.991 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.991 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:04:18.991 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:04:18.991 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.991 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:04:18.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:04:18.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:04:18.992 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:04:18.992 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.992 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.992 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.992 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:04:18.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:04:18.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:04:18.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:04:18.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.993 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.993 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:04:18.993 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:04:18.993 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.993 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.994 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.994 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:04:18.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:04:18.994 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.994 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:18.994 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.994 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.994 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.994 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:18.994 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:18.994 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 305.63031921700002, 2024-08-30T16:04:18.994 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:04:18.995 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:18.995 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:18.995 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:18.995 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:18.995 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:18.995 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:04:18.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.995 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:18.995 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.995 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:18.995 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:18.995 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.995 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:18.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:18.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:18.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:18.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:04:18.996 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:18.996 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.996 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.996 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:18.996 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:04:18.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:04:18.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:18.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:04:18.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:04:18.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.997 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:18.997 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:04:18.997 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:04:18.997 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.997 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:18.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:04:18.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:04:18.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:18.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:04:18.998 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:04:18.998 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.998 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.998 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:18.998 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:04:18.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:04:18.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:18.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:04:18.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:04:18.999 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:18.999 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:18.999 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:18.999 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:04:18.999 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:04:18.999 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:18.999 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:19.000 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.000 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.000 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.000 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:19.000 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:19.000 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 305.59658768499997, 2024-08-30T16:04:19.000 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:04:19.000 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:19.000 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:19.000 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:19.000 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:19.000 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:19.001 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:04:19.001 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.001 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:19.001 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:19.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:19.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.004 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.004 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.004 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:19.004 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:19.004 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:19.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:04:19.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:04:19.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.005 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:19.005 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:04:19.005 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:04:19.005 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.005 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:19.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:04:19.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:04:19.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:19.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:04:19.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:04:19.006 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.006 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.006 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:19.006 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:04:19.006 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:04:19.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:19.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:04:19.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:04:19.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:19.007 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:04:19.007 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:04:19.007 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.007 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.007 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:19.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:04:19.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:04:19.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.008 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:19.008 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:04:19.008 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:04:19.008 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.008 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:19.008 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.008 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.008 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.009 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:19.009 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:19.009 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 305.59606167700002, 2024-08-30T16:04:19.009 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:04:19.009 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:19.009 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:19.009 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:19.009 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:19.009 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:19.009 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:04:19.009 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.009 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:19.009 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:19.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:19.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:19.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:19.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.010 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.010 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.010 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:19.010 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:04:19.010 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:19.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:19.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:04:19.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:04:19.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:19.011 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:04:19.011 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:04:19.011 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.011 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.011 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:19.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:04:19.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:04:19.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:19.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:04:19.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:04:19.012 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.012 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.012 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:19.012 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:04:19.012 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:04:19.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:19.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:04:19.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:04:19.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:19.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:04:19.013 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:04:19.013 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.013 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.013 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:19.013 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:04:19.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:04:19.014 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.014 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:19.014 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.014 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.014 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.014 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:19.014 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:19.014 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 305.59560611400002, 2024-08-30T16:04:19.014 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:04:19.014 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:19.014 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:19.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:19.014 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:19.015 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:19.015 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:04:19.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.015 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:19.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:19.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:19.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.015 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.015 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.015 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:19.015 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:19.015 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:19.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:04:19.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:04:19.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.016 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:19.016 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:04:19.016 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:04:19.016 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.016 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:19.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:04:19.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:04:19.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:19.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:04:19.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:04:19.017 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.017 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.017 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:19.017 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:04:19.017 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:04:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:04:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:04:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:04:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:04:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.018 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:19.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:04:19.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:04:19.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.019 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:19.019 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:04:19.019 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:04:19.019 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.019 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:19.019 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.019 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.019 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.019 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:04:19.020 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:19.020 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 300.92509079799999, 2024-08-30T16:04:19.020 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:04:19.020 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:19.020 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:19.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:19.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:19.020 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:19.020 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:04:19.020 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.020 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:19.020 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.020 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:19.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:19.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:19.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:19.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.021 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:19.021 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:04:19.021 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:19.021 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.021 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:19.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:04:19.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:04:19.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:19.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:04:19.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:04:19.022 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.022 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.022 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:19.022 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:04:19.022 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:04:19.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:19.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:04:19.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:04:19.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:19.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:04:19.023 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:04:19.023 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.023 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.023 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:19.023 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:04:19.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:04:19.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:19.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:04:19.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:04:19.024 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.024 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.024 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:19.024 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:04:19.024 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:04:19.024 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.024 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:19.025 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.025 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.025 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:19.025 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:19.026 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:19.027 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:19.025+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:19.128 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:19.127+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:19.153 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:19.153 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:19.153 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:19.163 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:19.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '11 of 30' 2024-08-30T16:04:19.260 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:19.256+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:19.287 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:19.287 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:19.287 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:19.287 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:19.287 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:19.296 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:19.340 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:19.338+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:19.441 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.441 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 291.05537770799998, 2024-08-30T16:04:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:04:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:04:19.442 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.443 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:04:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:04:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:04:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:04:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:04:19.444 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:04:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:04:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:04:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:04:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:04:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:19.445 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:04:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:04:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:04:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 170.942381592, 2024-08-30T16:04:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:04:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:19.446 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:04:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:19.447 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:04:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:04:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:04:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.448 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:04:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:04:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:04:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:04:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:04:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:04:19.449 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:04:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:04:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-08-30T16:04:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 50.933962221000002, 2024-08-30T16:04:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037103309999999999, 2024-08-30T16:04:19.450 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 10 2024-08-30T16:04:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.451 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496549+0000", 2024-08-30T16:04:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496682+0000", 2024-08-30T16:04:19.452 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013371699999999999 2024-08-30T16:04:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496889+0000", 2024-08-30T16:04:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00020673600000000001 2024-08-30T16:04:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.497052+0000", 2024-08-30T16:04:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000162578 2024-08-30T16:04:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:19.453 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.498833+0000", 2024-08-30T16:04:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017810530000000001 2024-08-30T16:04:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.500282+0000", 2024-08-30T16:04:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001449437 2024-08-30T16:04:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(2.3 83/82 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 25'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@25'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:19.454 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 11.55291819, 2024-08-30T16:04:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088711689999999999, 2024-08-30T16:04:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.455 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877588+0000", 2024-08-30T16:04:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999723 2024-08-30T16:04:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877978+0000", 2024-08-30T16:04:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038947500000000002 2024-08-30T16:04:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.456 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.878456+0000", 2024-08-30T16:04:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047795699999999999 2024-08-30T16:04:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.878946+0000", 2024-08-30T16:04:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000490235 2024-08-30T16:04:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:19.457 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.880512+0000", 2024-08-30T16:04:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001565484 2024-08-30T16:04:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.880749+0000", 2024-08-30T16:04:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000237712 2024-08-30T16:04:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.886487+0000", 2024-08-30T16:04:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057377349999999999 2024-08-30T16:04:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.458 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(2.3 83/82 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-08-30T16:04:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 11.40551088, 2024-08-30T16:04:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062707819999999999, 2024-08-30T16:04:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:19.459 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025012+0000", 2024-08-30T16:04:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:04:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.460 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025103+0000", 2024-08-30T16:04:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0877000000000004e-05 2024-08-30T16:04:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025231+0000", 2024-08-30T16:04:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012737499999999999 2024-08-30T16:04:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025355+0000", 2024-08-30T16:04:19.461 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124194 2024-08-30T16:04:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.026750+0000", 2024-08-30T16:04:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001394922 2024-08-30T16:04:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.031294+0000", 2024-08-30T16:04:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045441270000000002 2024-08-30T16:04:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:19.462 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:19.463 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:19.463 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:19.463 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:19.439+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:19.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:19.532 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:19.531+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:19.607 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:19.608 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:19.608 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:19.608 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:19.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:19.667 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:19.664+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:19.734 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:19.732+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:19.748 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:19.748 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:19.748 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:19.748 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:19.748 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:19.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:19.834 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:19.832+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:19.933 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:19.933 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:19.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:19.934 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:19.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:19.934 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(2.3 83/82 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 25'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@25'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:19.934 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:19.934 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.033816941, 2024-08-30T16:04:19.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.10442118, 2024-08-30T16:04:19.934 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:19.934 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:19.934 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:19.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:19.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:19.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:19.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:19.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:19.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:19.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:19.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:19.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:19.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:19.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:19.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:19.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895800+0000", 2024-08-30T16:04:19.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999809 2024-08-30T16:04:19.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:19.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:19.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:19.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.921337+0000", 2024-08-30T16:04:19.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.02553677 2024-08-30T16:04:19.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:19.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:19.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:19.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.926424+0000", 2024-08-30T16:04:19.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050871340000000001 2024-08-30T16:04:19.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:19.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:19.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.935234+0000", 2024-08-30T16:04:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088098540000000006 2024-08-30T16:04:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.944321+0000", 2024-08-30T16:04:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0090873329999999995 2024-08-30T16:04:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.949613+0000", 2024-08-30T16:04:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052919710000000003 2024-08-30T16:04:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:19.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:19.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:19.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:08.000240+0000", 2024-08-30T16:04:19.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.050627411999999997 2024-08-30T16:04:19.938 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:19.938 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:19.938 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:19.938 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:19.938 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:19.938 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:19.940 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:19.938+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:19.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:20.062 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:20.059+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:20.076 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:20.077 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:20.077 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:20.077 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:20.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:20.142 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:20.141+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:20.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '12 of 30' 2024-08-30T16:04:20.243 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:20.242+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:20.261 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:20.262 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:20.262 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:20.262 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:20.262 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:20.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:20.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-08-30T16:04:20.357 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:20.355+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:20.444 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:20.442+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:20.451 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 299.17516071699998, 2024-08-30T16:04:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:04:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:20.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:20.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:04:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:04:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:04:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:04:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:04:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:04:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:04:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:04:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:04:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:04:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:04:20.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:04:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:04:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:04:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:20.456 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 298.90223335600001, 2024-08-30T16:04:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:04:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:20.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:04:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:04:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:04:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:04:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:04:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:04:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:04:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:04:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:04:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:04:20.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:04:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:04:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:04:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:04:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:20.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPull(2.3 e83/82 cost 1049576)", 2024-08-30T16:04:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.618147118, 2024-08-30T16:04:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051106587000000002, 2024-08-30T16:04:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:20.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825962+0000", 2024-08-30T16:04:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:20.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.833441+0000", 2024-08-30T16:04:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0074797580000000004 2024-08-30T16:04:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.833651+0000", 2024-08-30T16:04:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000209414 2024-08-30T16:04:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.834053+0000", 2024-08-30T16:04:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040233199999999998 2024-08-30T16:04:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.835125+0000", 2024-08-30T16:04:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010716930000000001 2024-08-30T16:04:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.877088+0000", 2024-08-30T16:04:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.041962869999999999 2024-08-30T16:04:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:20.464 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:20.465 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:20.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:20.512 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:04:20.552 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:20.548+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:20.636 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:20.636 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:20.636 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:20.636 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:20.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:20.646 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:20.644+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:20.747 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:20.746+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:20.780 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:20.781 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:20.781 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:20.781 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:20.781 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:20.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:20.825 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:20.848 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:20.846+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:20.935 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:20.936 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:20.948 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:20.946+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:20.965 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:20.965 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:20.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:20.965 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:20.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:04:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 295.58788620000001, 2024-08-30T16:04:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:04:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:04:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:04:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:04:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:04:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:20.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:04:20.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:04:20.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:20.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:04:20.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:04:20.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:20.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:04:20.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:04:20.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:20.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:04:20.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:04:20.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:20.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:04:20.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:04:20.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:20.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:20.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:20.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:04:20.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:04:20.970 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:20.970 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:20.970 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:20.970 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:20.970 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:20.970 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:20.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:21.028 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:21.028 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:21.050 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:21.047+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:21.136 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:21.136 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:21.159 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:21.157+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:21.181 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:21.181 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:21.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:21.238 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:21.239 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:21.251 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:21.250+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:21.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '13 of 30' 2024-08-30T16:04:21.308 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-08-30T16:04:21.323 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:21.358 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:21.356+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:21.385 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:21.385 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:21.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:21.413 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:21.413 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:21.417 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:21.417 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:21.460 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:21.457+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:21.525 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:21.525 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:21.535 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:21.577 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:21.574+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:21.655 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:21.653+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:21.660 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:21.661 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:21.670 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:21.755 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:21.753+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:21.834 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:21.834 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:21.843 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:21.855 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:21.853+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:21.957 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:21.955+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:21.970 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:21.970 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:21.979 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:22.058 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:22.055+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:22.111 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:22.111 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:22.121 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:22.159 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:22.157+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:22.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '14 of 30' 2024-08-30T16:04:22.247 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:22.247 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:22.257 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:22.285 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:22.283+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:22.378 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:22.376+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:22.387 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:22.387 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:22.396 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:22.461 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:22.459+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:22.562 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:22.560+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:22.578 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:22.579 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:22.589 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:22.663 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:22.661+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:22.714 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:22.714 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:22.723 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:22.764 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:22.762+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:22.864 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:22.862+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:22.892 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:22.892 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:22.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:22.966 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:22.964+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:23.030 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:23.030 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:23.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:23.081 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:23.079+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:23.167 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:23.165+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:23.211 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:23.211 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:23.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:23.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '15 of 30' 2024-08-30T16:04:23.269 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:23.267+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:23.323 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7] out_osds: [6, 4, 2] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-08-30T16:04:23.323 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:04:23.323 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 3, 5, 7] 2024-08-30T16:04:23.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-08-30T16:04:23.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-08-30T16:04:23.370 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:23.368+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:23.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:04:23.437 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:23.437 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:23.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:23.488 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:23.486+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:23.572 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:23.570+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:23.616 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:23.617 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:23.625 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:23.673 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:23.671+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:23.769 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:23.769 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:23.775 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:23.773+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:23.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:23.848 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:23.876 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:23.875+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:23.941 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:23.941 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:23.977 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:23.975+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:23.988 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 302.71121331500001, 2024-08-30T16:04:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:04:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:04:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:04:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:04:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:04:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:04:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:04:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:04:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:04:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:04:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:04:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:04:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:04:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:04:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:04:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 302.43828595399998, 2024-08-30T16:04:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:04:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:04:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:04:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:04:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:04:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:04:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:04:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:04:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:04:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:04:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:04:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:04:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:04:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:04:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:04:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPull(2.3 e83/82 cost 1049576)", 2024-08-30T16:04:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 16.154199716000001, 2024-08-30T16:04:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051106587000000002, 2024-08-30T16:04:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825962+0000", 2024-08-30T16:04:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.833441+0000", 2024-08-30T16:04:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0074797580000000004 2024-08-30T16:04:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.833651+0000", 2024-08-30T16:04:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000209414 2024-08-30T16:04:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.834053+0000", 2024-08-30T16:04:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040233199999999998 2024-08-30T16:04:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.835125+0000", 2024-08-30T16:04:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010716930000000001 2024-08-30T16:04:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.877088+0000", 2024-08-30T16:04:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.041962869999999999 2024-08-30T16:04:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:24.002 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:24.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:24.039 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:24.040 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:24.077 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:24.075+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:24.166 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:24.166 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:24.178 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:24.176+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:24.207 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:24.208 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:24.208 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:24.208 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:24.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:24.267 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:24.267 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:24.278 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:24.276+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:24.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '16 of 30' 2024-08-30T16:04:24.352 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T16:04:24.348+0000 1008f640 -1 mon.a@0(leader).osd e89 definitely_dead 0 2024-08-30T16:04:24.365 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:24.365 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:24.369 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:24.369 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:24.369 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:24.370 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:24.370 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:24.379 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:24.377+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:24.380 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:24.392 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:24.480 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:24.480 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:24.484 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:24.511 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:24.508+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:24.525 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:04:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 299.14812891999998, 2024-08-30T16:04:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:04:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:04:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:04:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:04:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:04:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:04:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:04:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:04:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:04:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:04:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:04:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:04:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:04:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:04:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:04:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:04:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:04:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:24.531 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:24.534 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:24.581 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:24.579+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:24.688 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:24.686+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:24.702 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:24.702 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:24.702 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:24.702 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:24.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:24.797 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:24.794+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:24.880 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:24.880 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:24.880 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:24.880 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:24.880 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:24.884 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:24.881+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:24.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:24.985 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:24.983+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:25.084 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:25.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 300.63150248800002, 2024-08-30T16:04:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:04:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:25.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:04:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:04:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:04:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:04:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:04:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:04:25.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:04:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:04:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:04:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:04:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:04:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:04:25.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:04:25.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:25.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:04:25.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:04:25.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:25.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:04:25.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:04:25.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:04:25.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:04:25.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:04:25.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:25.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:04:25.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:04:25.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:25.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:04:25.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:04:25.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:25.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:04:25.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:04:25.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:25.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:04:25.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:04:25.092 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.092 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:25.092 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.092 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:04:25.092 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:25.092 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 299.515636153, 2024-08-30T16:04:25.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:04:25.093 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:25.093 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:25.093 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:25.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:25.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:25.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:25.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:25.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:25.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:04:25.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:25.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:25.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:04:25.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:04:25.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:25.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:04:25.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:04:25.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:25.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:04:25.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:04:25.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:25.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:04:25.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:04:25.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:25.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:04:25.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:04:25.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:25.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:04:25.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:04:25.096 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.097 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:25.097 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.097 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:25.097 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:25.097 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 299.45604465500003, 2024-08-30T16:04:25.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:04:25.097 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:25.097 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:25.097 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:25.098 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:25.098 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:25.098 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:04:25.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.098 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:25.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:25.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:25.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:04:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:04:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:04:25.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:25.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:04:25.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:04:25.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:25.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:04:25.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:04:25.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:25.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:04:25.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:04:25.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:25.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:04:25.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:04:25.101 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.101 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:25.101 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.101 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:04:25.102 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:25.102 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 299.42934596800001, 2024-08-30T16:04:25.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:04:25.102 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:25.102 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:25.102 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:25.102 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:25.102 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:25.102 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:25.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.102 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:25.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:25.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:25.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:25.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:25.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:25.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:04:25.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:25.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:25.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:04:25.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:04:25.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:25.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:04:25.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:04:25.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:04:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:04:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:04:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:04:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:04:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:04:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.105 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:25.106 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.106 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:25.106 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:25.106 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 298.56172944000002, 2024-08-30T16:04:25.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:04:25.106 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:25.106 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:25.106 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:25.106 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:25.106 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:25.106 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:25.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.107 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:25.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:25.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:25.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:25.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:25.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:25.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:04:25.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:25.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:25.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:04:25.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:04:25.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:25.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:04:25.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:04:25.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:25.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:04:25.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:04:25.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:25.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:04:25.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:04:25.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:25.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:04:25.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:04:25.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:25.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:04:25.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:04:25.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:25.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:04:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:04:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:04:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:04:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:04:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:04:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:04:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:04:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:04:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 298.40248236100001, 2024-08-30T16:04:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:04:25.112 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:04:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:04:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:04:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:04:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:04:25.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:04:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:04:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:04:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:04:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:04:25.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:04:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:04:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:04:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:04:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.116 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 298.126308295, 2024-08-30T16:04:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:04:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:25.117 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:25.118 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:25.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.118 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:25.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:25.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:25.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:04:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:25.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:04:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:04:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:04:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:04:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:04:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:04:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:04:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:04:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:04:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:04:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 297.46801671600002, 2024-08-30T16:04:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:04:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:04:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.122 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:25.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:04:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:04:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:04:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:04:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:04:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:04:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:04:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:04:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:04:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:04:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:04:25.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:04:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:04:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:04:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:04:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:04:25.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:04:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:04:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:04:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:04:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:04:25.127 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:04:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 297.37210343200002, 2024-08-30T16:04:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:04:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:04:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:25.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:04:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:04:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:04:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:04:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:25.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:04:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:04:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:04:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:04:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:04:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:04:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:04:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:04:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:25.132 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 238.12074292099999, 2024-08-30T16:04:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:04:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:04:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:04:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:25.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:04:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:04:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:04:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:04:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:04:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:04:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:04:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:04:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:04:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:04:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.136 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:04:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 178.111638234, 2024-08-30T16:04:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:04:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.137 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:04:25.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:04:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:04:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:04:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:04:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:04:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:04:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:04:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:04:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:04:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:04:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:04:25.140 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.141 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:25.141 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.141 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:25.141 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:25.141 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:25.142 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:25.143 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:25.141+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:25.244 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:25.241+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:25.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '17 of 30' 2024-08-30T16:04:25.275 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:25.275 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:25.275 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:25.275 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:25.284 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:25.344 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:25.342+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:25.356 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-08-30T16:04:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:marked out osd.5. 2024-08-30T16:04:25.412 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:25.413 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:25.413 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:25.422 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:25.464 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:25.462+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:25.546 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:25.543+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:25.603 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:25.603 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:04:25.603 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:25.612 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:25.647 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:25.645+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:25.734 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:25.734 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:25.734 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:25.735 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:25.744 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:25.747 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:25.745+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:25.848 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:25.846+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:25.865 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:25.865 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:25.866 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:25.875 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:25.949 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:25.946+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:26.050 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.051 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 315.18679683900001, 2024-08-30T16:04:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:04:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.052 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:04:26.053 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.054 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.055 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.055 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:26.055 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.055 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.055 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.056 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.056 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:04:26.056 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:04:26.056 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.057 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.057 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.057 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:04:26.058 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:04:26.058 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.058 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.058 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.059 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:04:26.059 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:04:26.059 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.059 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.059 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:04:26.060 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:04:26.060 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.060 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.060 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.060 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:04:26.061 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:04:26.061 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.061 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.061 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:04:26.061 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:04:26.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:04:26.062 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.062 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.062 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.062 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:04:26.062 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:04:26.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.063 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.063 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.063 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:04:26.063 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:04:26.063 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.064 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.064 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:04:26.064 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:04:26.064 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.064 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.065 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.065 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:04:26.065 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:04:26.066 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.066 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.066 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.066 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:04:26.066 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:04:26.066 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.067 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.067 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.067 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.067 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:26.067 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:26.068 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.27038081900002, 2024-08-30T16:04:26.068 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:04:26.068 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.068 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.069 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.069 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.069 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.069 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:04:26.069 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.070 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.070 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.070 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.070 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:26.070 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.071 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.071 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:26.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.071 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:04:26.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:26.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:04:26.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:04:26.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:04:26.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:04:26.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:04:26.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:04:26.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:04:26.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:04:26.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:04:26.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:04:26.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:04:26.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:04:26.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:04:26.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:04:26.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.078 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.078 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:04:26.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:04:26.079 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.079 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.079 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.080 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:26.080 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:26.080 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 314.268676266, 2024-08-30T16:04:26.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:04:26.081 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.081 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.081 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.081 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.081 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.081 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:04:26.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.082 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:26.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:26.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:04:26.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:04:26.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:04:26.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:04:26.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:04:26.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:04:26.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:04:26.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:04:26.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.089 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.089 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:04:26.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:04:26.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:04:26.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:04:26.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:04:26.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:04:26.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:04:26.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:04:26.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:04:26.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:04:26.092 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.092 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.092 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.092 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.093 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:26.093 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 313.84922245899998, 2024-08-30T16:04:26.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:04:26.093 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.093 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.093 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.094 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:04:26.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.094 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.094 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:26.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.094 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:26.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:04:26.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:04:26.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:04:26.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:04:26.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:04:26.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:04:26.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:04:26.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:04:26.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:04:26.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:04:26.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:04:26.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:04:26.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:04:26.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:04:26.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:04:26.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:04:26.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:04:26.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:04:26.099 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.099 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.099 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.100 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.100 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:26.100 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 313.832334285, 2024-08-30T16:04:26.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:04:26.100 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.100 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.100 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:04:26.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.101 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:26.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:26.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:04:26.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:04:26.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:04:26.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:04:26.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:04:26.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:04:26.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:04:26.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:04:26.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:04:26.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:04:26.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:04:26.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:04:26.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:04:26.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:04:26.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:04:26.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:04:26.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:04:26.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:04:26.105 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.105 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.105 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.105 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.105 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:26.105 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 313.81933421600002, 2024-08-30T16:04:26.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:04:26.105 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.105 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.106 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.106 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.106 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:04:26.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.106 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:26.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:26.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:04:26.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:04:26.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:04:26.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:04:26.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:04:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:04:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:04:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:04:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:04:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:04:26.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:04:26.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:04:26.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:04:26.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:04:26.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:04:26.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:04:26.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.110 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:04:26.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:04:26.110 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.111 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.111 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.111 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.111 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:26.111 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 313.81684192400002, 2024-08-30T16:04:26.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:04:26.111 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.111 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.111 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.111 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.111 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.112 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:04:26.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.112 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:26.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:26.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:04:26.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:26.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:04:26.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:04:26.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:04:26.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:04:26.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:04:26.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:04:26.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:04:26.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:04:26.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:04:26.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:04:26.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:04:26.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:04:26.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:04:26.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:04:26.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.116 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.116 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:04:26.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:04:26.116 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.117 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.117 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.117 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.117 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:26.117 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 313.81608447600001, 2024-08-30T16:04:26.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:04:26.117 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.117 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.117 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.117 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.117 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.117 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:04:26.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.118 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:26.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:26.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:04:26.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:04:26.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:04:26.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:04:26.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:04:26.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:04:26.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:04:26.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:04:26.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:04:26.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:04:26.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:04:26.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:04:26.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:04:26.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:04:26.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.121 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:04:26.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:04:26.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:04:26.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:04:26.122 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.122 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.122 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.122 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.122 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:26.123 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 313.81535202600003, 2024-08-30T16:04:26.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:04:26.123 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.123 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.123 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.123 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.123 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.123 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:04:26.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.123 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:26.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:26.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:04:26.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:26.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:04:26.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:04:26.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:04:26.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:04:26.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:04:26.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:04:26.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:04:26.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:04:26.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:04:26.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:04:26.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:04:26.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:04:26.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:04:26.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:04:26.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.127 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.127 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:04:26.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:04:26.127 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.127 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.128 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.128 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.128 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:26.128 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 313.81281322699999, 2024-08-30T16:04:26.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:04:26.128 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.128 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.128 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.128 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.128 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.128 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:04:26.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.129 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:26.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:26.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:04:26.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:26.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:04:26.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:04:26.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:04:26.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:04:26.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:04:26.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:04:26.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:04:26.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:04:26.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:04:26.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:04:26.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:04:26.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:04:26.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:04:26.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:04:26.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:04:26.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:04:26.133 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.133 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.133 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.133 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.133 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:26.133 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 313.81206522600002, 2024-08-30T16:04:26.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:04:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:04:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:26.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:04:26.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:04:26.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:04:26.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:04:26.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:04:26.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:04:26.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:04:26.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:04:26.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:04:26.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:04:26.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:04:26.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:04:26.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:04:26.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:04:26.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:04:26.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:04:26.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.138 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.138 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:04:26.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:04:26.138 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.138 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.139 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.139 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.139 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:26.139 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 313.03994875400002, 2024-08-30T16:04:26.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:04:26.139 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.139 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.139 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.139 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.139 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.139 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:04:26.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.140 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:26.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:26.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:04:26.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:26.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:04:26.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:04:26.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:04:26.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:04:26.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:04:26.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:04:26.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:04:26.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:04:26.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:04:26.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:04:26.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:04:26.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:04:26.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:04:26.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:04:26.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:04:26.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:04:26.147 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.147 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.147 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.148 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.148 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:26.148 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 313.037891608, 2024-08-30T16:04:26.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:04:26.149 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.149 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.149 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.149 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.149 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.149 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:04:26.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.149 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:26.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:26.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:04:26.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:26.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:04:26.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:04:26.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:04:26.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:04:26.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:04:26.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:04:26.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:04:26.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:04:26.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:04:26.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:04:26.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:04:26.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:04:26.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:04:26.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:04:26.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:04:26.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:04:26.153 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.153 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.153 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.154 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.154 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:26.154 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 312.77494710799999, 2024-08-30T16:04:26.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:04:26.154 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.154 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.154 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.154 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.154 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.154 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:04:26.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.154 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:26.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:26.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:04:26.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:04:26.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:04:26.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:04:26.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:04:26.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:04:26.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:04:26.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:04:26.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:04:26.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:04:26.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:04:26.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:04:26.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:04:26.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:04:26.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:04:26.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:04:26.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:04:26.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:04:26.159 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.159 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.159 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.159 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.159 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:26.159 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 312.77385560499999, 2024-08-30T16:04:26.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:04:26.159 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.159 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.159 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.160 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.160 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.160 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:04:26.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.160 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:26.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:26.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:04:26.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:26.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:04:26.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:04:26.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:04:26.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:04:26.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:04:26.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:04:26.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:04:26.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:04:26.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:04:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:04:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:04:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:04:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:04:26.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:04:26.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:04:26.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:04:26.164 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.164 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.164 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.164 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.164 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:26.164 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 312.77310602, 2024-08-30T16:04:26.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:04:26.165 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.165 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.165 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.165 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.165 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.165 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:04:26.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.165 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:26.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:26.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.166 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:04:26.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:26.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:04:26.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:04:26.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:04:26.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:04:26.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:04:26.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:04:26.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:04:26.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:04:26.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:04:26.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:04:26.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:04:26.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:04:26.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:04:26.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:04:26.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:04:26.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:04:26.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.170 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.170 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.170 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.170 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:26.170 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 312.73937448800001, 2024-08-30T16:04:26.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:04:26.170 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.170 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.170 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:04:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:26.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:04:26.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:04:26.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:04:26.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:04:26.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:04:26.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:04:26.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:04:26.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:04:26.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:04:26.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:04:26.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:04:26.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:04:26.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:04:26.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:04:26.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:04:26.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:04:26.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:04:26.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:04:26.175 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.175 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.175 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.175 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.175 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:26.175 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 312.73884848, 2024-08-30T16:04:26.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:04:26.176 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.176 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.176 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.176 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.176 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.176 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:04:26.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.176 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:26.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:26.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:04:26.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:26.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:04:26.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:04:26.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:04:26.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:04:26.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:04:26.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:04:26.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:04:26.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:04:26.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:04:26.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:04:26.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:04:26.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:04:26.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:04:26.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:04:26.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:04:26.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:04:26.180 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.180 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.180 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.181 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:26.181 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:26.181 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 312.738392917, 2024-08-30T16:04:26.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:04:26.181 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.181 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.181 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.181 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.181 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.181 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:04:26.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.181 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:26.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:26.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.182 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.182 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.182 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.182 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:04:26.182 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:04:26.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:04:26.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:04:26.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.183 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:04:26.183 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:04:26.183 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.183 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.183 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:04:26.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:04:26.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:04:26.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:04:26.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.184 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.184 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.184 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:04:26.184 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:04:26.184 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:04:26.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:04:26.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:04:26.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:04:26.185 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.185 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.185 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.185 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:04:26.185 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:04:26.186 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.186 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.186 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.186 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.186 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.186 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:04:26.186 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:26.186 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 308.06787760100002, 2024-08-30T16:04:26.186 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:04:26.186 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.186 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:26.186 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.186 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:26.186 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:26.187 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:04:26.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.187 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:26.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.187 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.187 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.187 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:26.187 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.187 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:04:26.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:26.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:04:26.188 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:04:26.188 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.188 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.188 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.188 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:04:26.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:04:26.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:04:26.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:04:26.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.189 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:04:26.189 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:04:26.189 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.189 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.189 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:04:26.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:04:26.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:26.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:04:26.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:04:26.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.190 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.190 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:26.190 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:04:26.190 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:04:26.190 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.191 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.191 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.191 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:04:26.191 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:04:26.191 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.191 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.191 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.191 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.191 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.191 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:26.192 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:26.048+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:26.193 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:26.193 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:26.191+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:26.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '18 of 30' 2024-08-30T16:04:26.314 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:26.312+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:26.358 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:26.359 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:26.359 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:26.369 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:26.394 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:26.392+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:26.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:04:26.540 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:26.538+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:26.606 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:26.602+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:26.630 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:26.630 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:26.631 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:26.631 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:26.631 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:26.640 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:26.701 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:26.698+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:26.766 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:26.766 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:26.791 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 298.406107491, 2024-08-30T16:04:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:04:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:26.792 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:04:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:04:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:04:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:04:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:04:26.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:04:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:04:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:04:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:04:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.795 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:04:26.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:04:26.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:04:26.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:04:26.796 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.796 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.796 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:04:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 178.293111375, 2024-08-30T16:04:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:04:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.797 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:04:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:26.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:04:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:26.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:04:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:04:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:04:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:04:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.800 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:04:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:04:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:04:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:04:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:04:26.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:04:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-08-30T16:04:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 58.284692004, 2024-08-30T16:04:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037103309999999999, 2024-08-30T16:04:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.802 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 10 2024-08-30T16:04:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496549+0000", 2024-08-30T16:04:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496682+0000", 2024-08-30T16:04:26.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013371699999999999 2024-08-30T16:04:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496889+0000", 2024-08-30T16:04:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00020673600000000001 2024-08-30T16:04:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.497052+0000", 2024-08-30T16:04:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000162578 2024-08-30T16:04:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.498833+0000", 2024-08-30T16:04:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017810530000000001 2024-08-30T16:04:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.500282+0000", 2024-08-30T16:04:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001449437 2024-08-30T16:04:26.806 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(2.3 83/82 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 25'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@25'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.903647972999998, 2024-08-30T16:04:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088711689999999999, 2024-08-30T16:04:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877588+0000", 2024-08-30T16:04:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999723 2024-08-30T16:04:26.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877978+0000", 2024-08-30T16:04:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038947500000000002 2024-08-30T16:04:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.878456+0000", 2024-08-30T16:04:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047795699999999999 2024-08-30T16:04:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.878946+0000", 2024-08-30T16:04:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000490235 2024-08-30T16:04:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.880512+0000", 2024-08-30T16:04:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001565484 2024-08-30T16:04:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.880749+0000", 2024-08-30T16:04:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000237712 2024-08-30T16:04:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.886487+0000", 2024-08-30T16:04:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057377349999999999 2024-08-30T16:04:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.812 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(2.3 83/82 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-08-30T16:04:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 18.756240663, 2024-08-30T16:04:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062707819999999999, 2024-08-30T16:04:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:26.813 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025012+0000", 2024-08-30T16:04:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:04:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025103+0000", 2024-08-30T16:04:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0877000000000004e-05 2024-08-30T16:04:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025231+0000", 2024-08-30T16:04:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012737499999999999 2024-08-30T16:04:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025355+0000", 2024-08-30T16:04:26.815 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124194 2024-08-30T16:04:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.026750+0000", 2024-08-30T16:04:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001394922 2024-08-30T16:04:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.031294+0000", 2024-08-30T16:04:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045441270000000002 2024-08-30T16:04:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.816 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:26.817 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:26.817 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:26.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:26.819 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:26.817+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:26.919 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:26.917+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:26.931 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:26.931 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:26.965 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:26.966 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:26.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:27.020 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:27.017+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:27.048 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:27.048 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:27.107 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:27.107 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:27.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:27.120 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:27.119+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:27.140 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:27.141 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:27.226 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:27.226 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:27.227 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:27.224+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:27.282 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:27.282 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:27.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:27.301 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '19 of 30' 2024-08-30T16:04:27.309 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:27.310 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:27.312 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:27.322 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:27.320+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:27.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-08-30T16:04:27.392 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:27.393 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:27.397 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:27.398 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:27.426 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:27.426 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:27.427 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:27.425+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:27.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:27.524 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:27.522+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:27.609 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:27.609 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:27.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:27.625 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:27.623+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:27.726 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:27.725+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:27.746 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:27.747 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:27.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:27.829 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:27.828+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:27.886 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:27.887 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:27.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:27.941 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:27.938+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:28.027 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:28.028 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:28.031 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:28.029+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:28.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:28.132 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:28.130+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:28.206 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:28.207 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:28.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:28.259 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:28.255+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:28.334 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:28.332+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:28.391 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:28.392 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:28.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:28.414 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-08-30T16:04:28.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '20 of 30' 2024-08-30T16:04:28.435 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:28.433+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:28.536 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:28.534+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:28.549 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:28.549 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:28.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:28.643 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:28.641+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:28.727 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:28.727 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:28.737 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:28.777 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:28.776+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:28.839 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:28.838+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:28.863 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:28.864 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:28.873 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:28.941 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:28.938+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:29.039 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:29.039 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:29.041 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:29.039+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:29.048 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:29.165 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:29.163+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:29.172 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:29.172 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:29.181 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:29.244 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:29.242+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:29.345 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:29.343+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:29.350 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:29.350 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:29.362 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:29.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:04:29.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '21 of 30' 2024-08-30T16:04:29.465 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:29.462+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:29.532 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:29.532 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:29.542 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:29.547 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:29.546+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:29.638 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:29.638 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:29.655 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:29.654+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:29.669 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:29.669 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:29.678 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:29.728 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:29.729 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:29.748 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:29.747+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:29.821 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:29.821 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:29.850 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:29.848+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:29.859 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:29.860 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:29.868 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:29.925 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:29.926 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:29.953 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:29.951+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:30.010 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:30.010 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:30.037 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:30.037 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:30.046 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:30.051 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:30.050+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:30.093 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:30.094 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:30.096 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:30.181 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:30.179+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:30.181 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:30.182 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:30.185 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:30.230 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:30.230 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:30.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:30.230 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:30.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.230 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:30.230 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:30.230 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 301.840116266, 2024-08-30T16:04:30.230 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:04:30.231 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:30.231 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:30.231 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:30.231 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:30.231 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:30.231 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:04:30.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.231 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:30.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.231 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:30.231 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:30.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:30.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:30.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:30.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:30.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:30.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:04:30.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:04:30.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.232 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:30.232 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:04:30.232 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:04:30.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:30.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:04:30.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:04:30.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.233 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:30.233 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:04:30.233 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:04:30.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:30.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:04:30.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:04:30.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.234 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:30.234 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:04:30.234 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:04:30.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.234 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:30.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.235 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:04:30.235 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:30.235 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 181.72712014999999, 2024-08-30T16:04:30.235 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:04:30.235 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:30.235 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:30.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:30.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:30.235 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:30.235 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:04:30.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.235 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:30.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:30.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:30.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:30.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:30.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:30.236 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:30.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.236 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:30.236 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:04:30.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:30.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:30.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:04:30.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:04:30.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.237 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:30.237 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:04:30.237 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:04:30.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:30.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:04:30.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:04:30.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:30.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:04:30.238 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:04:30.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.238 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:30.238 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:04:30.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:04:30.239 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.239 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:30.239 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.239 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-08-30T16:04:30.239 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:30.239 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 61.718700779000002, 2024-08-30T16:04:30.239 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037103309999999999, 2024-08-30T16:04:30.239 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:30.239 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:30.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:30.239 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:30.240 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:30.240 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 10 2024-08-30T16:04:30.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.240 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:30.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:30.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:30.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:30.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.240 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:30.240 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:30.240 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:30.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:30.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496549+0000", 2024-08-30T16:04:30.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:30.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.241 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:30.241 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496682+0000", 2024-08-30T16:04:30.241 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013371699999999999 2024-08-30T16:04:30.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:30.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496889+0000", 2024-08-30T16:04:30.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00020673600000000001 2024-08-30T16:04:30.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:30.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.497052+0000", 2024-08-30T16:04:30.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000162578 2024-08-30T16:04:30.242 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.242 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.242 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:30.242 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.498833+0000", 2024-08-30T16:04:30.242 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017810530000000001 2024-08-30T16:04:30.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.243 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:30.243 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.500282+0000", 2024-08-30T16:04:30.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001449437 2024-08-30T16:04:30.243 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.243 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:30.243 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.243 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.243 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.243 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(2.3 83/82 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 25'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@25'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:30.243 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:30.243 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.337656748000001, 2024-08-30T16:04:30.243 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088711689999999999, 2024-08-30T16:04:30.244 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:30.244 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:30.244 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:30.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:30.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:30.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:30.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.244 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.244 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:30.244 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:30.244 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:30.244 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:30.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877588+0000", 2024-08-30T16:04:30.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999723 2024-08-30T16:04:30.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:30.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877978+0000", 2024-08-30T16:04:30.245 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038947500000000002 2024-08-30T16:04:30.245 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.245 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.245 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:30.245 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.878456+0000", 2024-08-30T16:04:30.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047795699999999999 2024-08-30T16:04:30.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:30.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.878946+0000", 2024-08-30T16:04:30.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000490235 2024-08-30T16:04:30.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:30.246 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.880512+0000", 2024-08-30T16:04:30.246 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001565484 2024-08-30T16:04:30.246 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.246 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.246 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:30.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.880749+0000", 2024-08-30T16:04:30.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000237712 2024-08-30T16:04:30.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.247 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:30.247 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.886487+0000", 2024-08-30T16:04:30.247 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057377349999999999 2024-08-30T16:04:30.247 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.247 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:30.247 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.247 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.247 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.247 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(2.3 83/82 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-08-30T16:04:30.247 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:30.248 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.190249437999999, 2024-08-30T16:04:30.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062707819999999999, 2024-08-30T16:04:30.248 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:30.248 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:30.248 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:30.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:30.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:30.248 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:30.248 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.248 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.248 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:30.248 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:30.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:30.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:30.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025012+0000", 2024-08-30T16:04:30.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:04:30.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.249 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:30.249 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025103+0000", 2024-08-30T16:04:30.249 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0877000000000004e-05 2024-08-30T16:04:30.249 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.249 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025231+0000", 2024-08-30T16:04:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012737499999999999 2024-08-30T16:04:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025355+0000", 2024-08-30T16:04:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124194 2024-08-30T16:04:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.026750+0000", 2024-08-30T16:04:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001394922 2024-08-30T16:04:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.251 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:30.251 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.031294+0000", 2024-08-30T16:04:30.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045441270000000002 2024-08-30T16:04:30.251 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.251 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:30.251 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.251 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.251 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.251 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.0 91/90 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:30.251 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:26.829025+0000", 2024-08-30T16:04:30.251 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.3862473500000001, 2024-08-30T16:04:30.251 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.060649472000000003, 2024-08-30T16:04:30.252 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:30.252 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:30.252 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:30.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.252 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:30.252 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.829025+0000", 2024-08-30T16:04:30.252 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:30.252 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.252 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:30.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.829025+0000", 2024-08-30T16:04:30.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:30.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:30.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.828966+0000", 2024-08-30T16:04:30.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999404 2024-08-30T16:04:30.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.253 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.253 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:30.253 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.830358+0000", 2024-08-30T16:04:30.253 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013920580000000001 2024-08-30T16:04:30.253 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:30.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.830913+0000", 2024-08-30T16:04:30.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00055505300000000001 2024-08-30T16:04:30.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:30.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.831281+0000", 2024-08-30T16:04:30.254 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000368193 2024-08-30T16:04:30.254 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.254 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.254 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:30.254 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.832628+0000", 2024-08-30T16:04:30.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001346863 2024-08-30T16:04:30.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:30.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.833344+0000", 2024-08-30T16:04:30.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071610100000000002 2024-08-30T16:04:30.255 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.255 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.255 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:30.255 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.889675+0000", 2024-08-30T16:04:30.255 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056330587000000001 2024-08-30T16:04:30.255 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.255 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:30.255 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.256 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.256 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.0 91/90 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:30.256 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:27.013394+0000", 2024-08-30T16:04:30.256 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 3.2018788709999999, 2024-08-30T16:04:30.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.021968463000000001, 2024-08-30T16:04:30.256 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:30.256 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:30.256 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:30.256 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.256 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:30.256 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.013394+0000", 2024-08-30T16:04:30.256 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:30.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:30.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.013394+0000", 2024-08-30T16:04:30.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:30.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:30.257 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.013362+0000", 2024-08-30T16:04:30.257 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999681 2024-08-30T16:04:30.257 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.257 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.257 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:30.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.017776+0000", 2024-08-30T16:04:30.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044143150000000003 2024-08-30T16:04:30.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:30.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.018020+0000", 2024-08-30T16:04:30.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000244076 2024-08-30T16:04:30.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.258 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:30.258 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.018340+0000", 2024-08-30T16:04:30.258 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00032027100000000001 2024-08-30T16:04:30.258 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.258 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:30.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.019618+0000", 2024-08-30T16:04:30.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012770640000000001 2024-08-30T16:04:30.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:30.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.019851+0000", 2024-08-30T16:04:30.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023378 2024-08-30T16:04:30.259 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:30.259 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:30.259 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:30.259 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.035362+0000", 2024-08-30T16:04:30.259 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015510896999999999 2024-08-30T16:04:30.259 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.260 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:30.260 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.260 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:30.260 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:30.260 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:30.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:30.262 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:30.260+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:30.362 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:30.361+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:30.382 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 7] out_osds: [6, 4, 2, 5] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-08-30T16:04:30.382 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:04:30.382 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-08-30T16:04:30.382 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T16:04:30.382 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T16:04:30.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-08-30T16:04:30.392 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:30.392 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:30.392 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:30.392 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:30.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:30.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-08-30T16:04:30.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '22 of 30' 2024-08-30T16:04:30.465 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:30.463+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:30.538 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:30.538 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-30T16:04:30.539 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:30.564 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:30.563+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:30.625 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:30.626 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:30.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:30.665 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:30.663+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:30.766 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:30.764+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:30.783 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:30.783 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:30.783 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:30.783 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:30.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:30.783 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(2.3 83/82 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 25'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@25'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:30.783 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:30.783 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 22.883685080999999, 2024-08-30T16:04:30.783 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.10442118, 2024-08-30T16:04:30.783 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:30.784 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:30.784 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:30.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:30.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:30.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:30.784 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:30.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:30.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:30.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:30.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:30.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:30.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:30.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:30.785 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:30.785 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895800+0000", 2024-08-30T16:04:30.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999809 2024-08-30T16:04:30.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:30.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:30.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:30.786 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.921337+0000", 2024-08-30T16:04:30.786 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.02553677 2024-08-30T16:04:30.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:30.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:30.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:30.786 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.926424+0000", 2024-08-30T16:04:30.786 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050871340000000001 2024-08-30T16:04:30.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:30.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:30.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:30.786 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.935234+0000", 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088098540000000006 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.944321+0000", 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0090873329999999995 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.949613+0000", 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052919710000000003 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:30.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:08.000240+0000", 2024-08-30T16:04:30.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.050627411999999997 2024-08-30T16:04:30.788 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:30.788 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:30.788 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:30.788 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:30.788 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:30.788 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:30.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:30.867 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:30.865+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:30.924 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:30.924 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:30.924 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:30.924 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:30.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:30.979 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:30.978+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:31.065 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:31.065 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:31.065 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:31.065 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:31.065 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:31.069 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:31.067+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:31.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:31.169 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:31.168+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:31.250 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:31.250 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:31.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:31.250 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:31.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.251 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:31.251 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:31.251 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 309.97367452899999, 2024-08-30T16:04:31.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:04:31.252 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:31.252 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:31.252 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:31.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:31.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:31.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:31.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:31.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:31.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:31.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:31.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:04:31.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:04:31.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:31.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:04:31.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:04:31.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:31.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:04:31.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:04:31.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:31.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:04:31.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:04:31.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:31.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:04:31.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:04:31.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:31.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:04:31.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:04:31.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:31.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:04:31.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:04:31.256 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:31.256 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:31.256 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:31.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.257 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:31.257 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:31.257 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 309.70074716800002, 2024-08-30T16:04:31.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:04:31.257 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:31.257 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:31.257 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:31.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:31.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:31.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:31.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:31.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:31.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:31.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:31.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:04:31.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:04:31.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:31.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:04:31.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:04:31.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:31.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:04:31.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:04:31.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.260 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:31.260 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:04:31.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:04:31.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.260 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:31.260 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:04:31.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:04:31.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.260 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:31.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:04:31.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:04:31.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:31.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:04:31.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:04:31.261 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:31.261 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:31.261 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:31.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.261 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPull(2.3 e83/82 cost 1049576)", 2024-08-30T16:04:31.262 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:31.262 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 23.416660929999999, 2024-08-30T16:04:31.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051106587000000002, 2024-08-30T16:04:31.262 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:31.262 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:31.262 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:31.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.262 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:31.262 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:31.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:31.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.262 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:31.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:31.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:31.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:31.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825962+0000", 2024-08-30T16:04:31.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:31.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:31.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.833441+0000", 2024-08-30T16:04:31.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0074797580000000004 2024-08-30T16:04:31.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.264 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:31.264 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.833651+0000", 2024-08-30T16:04:31.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000209414 2024-08-30T16:04:31.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.264 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:31.264 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.834053+0000", 2024-08-30T16:04:31.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040233199999999998 2024-08-30T16:04:31.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.265 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:31.265 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.835125+0000", 2024-08-30T16:04:31.265 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010716930000000001 2024-08-30T16:04:31.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.265 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:31.265 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.877088+0000", 2024-08-30T16:04:31.265 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.041962869999999999 2024-08-30T16:04:31.265 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:31.265 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:31.265 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:31.265 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:31.266 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:31.266 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:31.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:31.270 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:31.268+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:31.370 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:31.369+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:31.394 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:31.395 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:31.395 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:31.395 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:31.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:31.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '23 of 30' 2024-08-30T16:04:31.471 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:31.470+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:31.506 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-08-30T16:04:31.536 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:31.536 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:31.536 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:31.536 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:31.536 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:31.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:31.573 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:31.572+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:31.674 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:31.673+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:31.718 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:04:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 306.340518024, 2024-08-30T16:04:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:04:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:04:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:04:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:04:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:04:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:04:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:04:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:04:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:04:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:04:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:04:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:04:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:04:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:04:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:04:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:04:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:04:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:31.727 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:31.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:31.774 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:31.773+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:31.875 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:31.874+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:31.901 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:31.901 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:31.901 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:31.901 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:31.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:31.995 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:31.994+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:32.077 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:32.076+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:32.078 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:32.078 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:32.078 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:32.078 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:32.079 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:32.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:32.178 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:32.177+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:32.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:04:32.292 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 307.83608027100001, 2024-08-30T16:04:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:04:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:04:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:04:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:04:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:04:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:04:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:04:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:04:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:04:32.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:04:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:04:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:04:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:04:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:04:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:04:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:04:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:04:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:04:32.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:04:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:04:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:04:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:04:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:04:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:04:32.298 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:04:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:04:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:04:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:04:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:04:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.299 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:04:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 306.72021393599999, 2024-08-30T16:04:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:04:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:32.300 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:04:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.301 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:04:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:04:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:04:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:04:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:32.302 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:04:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:04:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:04:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:04:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:04:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:04:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:04:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:04:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:32.304 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 306.66062243800002, 2024-08-30T16:04:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:04:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:04:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:04:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:32.306 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:04:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:04:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:04:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:04:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:04:32.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:04:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:04:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:04:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:04:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:04:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:32.308 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:04:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 306.633923751, 2024-08-30T16:04:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:04:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:04:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:04:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:04:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:04:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:04:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:04:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:04:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:04:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:04:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:04:32.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 305.76630722300001, 2024-08-30T16:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:04:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:04:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:04:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:32.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:04:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:04:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:04:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:04:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:04:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:04:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:04:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:04:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:04:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:04:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:04:32.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:04:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:04:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:04:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:04:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:04:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:32.319 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:04:32.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:04:32.319 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.319 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:32.319 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.319 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:04:32.319 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:32.319 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 305.607060144, 2024-08-30T16:04:32.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:04:32.319 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:32.319 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:32.320 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:04:32.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:04:32.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.321 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:32.321 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:04:32.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:04:32.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.321 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:32.321 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:04:32.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:04:32.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:32.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:04:32.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:04:32.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:32.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:04:32.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:04:32.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:32.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:04:32.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:04:32.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.323 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:32.323 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:04:32.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:04:32.323 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.323 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:32.323 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.323 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:32.323 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:32.323 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 305.33088607799999, 2024-08-30T16:04:32.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:04:32.324 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:32.324 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:32.324 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:32.324 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:32.324 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:32.324 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:32.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.324 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:32.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.324 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:32.324 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:32.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.325 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:32.325 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:32.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.325 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:32.325 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:04:32.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:32.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.325 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:32.325 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:04:32.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:04:32.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:32.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:04:32.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:04:32.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:32.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:04:32.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:04:32.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:32.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:04:32.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:04:32.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:32.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:04:32.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:04:32.327 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.327 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:32.327 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.328 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:32.328 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:32.328 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 304.67259449900001, 2024-08-30T16:04:32.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:04:32.328 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:32.328 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:32.328 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:32.328 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:32.328 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:32.328 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:04:32.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.328 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:32.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:04:32.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:32.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:32.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:04:32.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:04:32.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:32.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:04:32.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:04:32.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:32.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:04:32.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:04:32.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:32.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:04:32.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:04:32.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:32.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:04:32.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:04:32.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:32.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:04:32.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:04:32.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:32.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:04:32.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:04:32.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:32.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:04:32.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:04:32.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:32.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:04:32.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:04:32.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:32.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:04:32.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:04:32.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.333 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:32.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.334 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:04:32.334 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:32.334 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 304.57668121500001, 2024-08-30T16:04:32.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:04:32.334 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:32.334 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:32.334 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:32.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:32.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:32.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:32.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:32.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:32.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:04:32.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:32.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:32.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:04:32.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:04:32.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:32.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:04:32.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:04:32.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:32.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:04:32.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:04:32.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:32.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:04:32.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:04:32.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:32.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:04:32.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:04:32.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:04:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:04:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 245.32532070400001, 2024-08-30T16:04:32.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:04:32.342 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:32.342 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:32.342 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:32.342 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:32.342 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:32.342 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:32.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.342 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:32.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:32.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:32.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:32.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:32.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:32.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:04:32.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:04:32.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:32.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:04:32.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:04:32.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:32.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:04:32.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:04:32.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:32.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:04:32.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:04:32.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:32.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:04:32.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:04:32.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:32.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:04:32.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:04:32.345 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.345 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:32.345 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.346 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:04:32.346 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:32.346 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 185.31621601699999, 2024-08-30T16:04:32.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:04:32.346 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:32.346 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:32.346 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:32.346 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:32.346 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:32.346 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:32.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.347 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:32.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:32.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:32.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:32.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:32.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:32.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:04:32.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:04:32.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:32.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:04:32.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:04:32.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:32.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:04:32.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:04:32.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:32.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:04:32.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:04:32.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:32.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:04:32.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:04:32.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:04:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:04:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:11 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-08-30T16:04:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:27.021982+0000", 2024-08-30T16:04:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.2408327420000003, 2024-08-30T16:04:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0095695959999999997, 2024-08-30T16:04:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:32.351 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:32.351 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:32.351 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:32.351 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 11 2024-08-30T16:04:32.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.351 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:32.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:32.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.021982+0000", 2024-08-30T16:04:32.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:32.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.021982+0000", 2024-08-30T16:04:32.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:32.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:32.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.021955+0000", 2024-08-30T16:04:32.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999728 2024-08-30T16:04:32.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:32.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.022110+0000", 2024-08-30T16:04:32.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015461899999999999 2024-08-30T16:04:32.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:32.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.022347+0000", 2024-08-30T16:04:32.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023690199999999999 2024-08-30T16:04:32.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:32.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.022532+0000", 2024-08-30T16:04:32.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000185339 2024-08-30T16:04:32.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:32.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.028853+0000", 2024-08-30T16:04:32.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063211270000000002 2024-08-30T16:04:32.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:32.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:32.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:32.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.031552+0000", 2024-08-30T16:04:32.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026985770000000002 2024-08-30T16:04:32.354 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.354 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:32.354 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.354 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:32.354 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:32.354 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:32.355 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:32.290+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:32.356 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:32.400 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:32.398+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:32.425 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:32.426 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:32.454 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '24 of 30' 2024-08-30T16:04:32.483 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:32.483 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:32.483 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:32.483 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:32.492 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:32.498 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:32.497+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:32.520 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:32.521 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:32.581 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:32.580+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:32.614 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:32.614 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:32.660 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:32.660 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:32.660 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:32.660 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:32.660 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:32.669 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:32.706 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:32.706 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:32.713 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:32.711+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:32.783 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:32.782+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:32.791 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:32.792 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:32.828 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:32.828 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:32.838 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:32.876 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:32.877 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:32.879 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:32.922 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:32.920+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:32.981 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:32.981 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:32.985 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:32.985 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:32.991 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:32.990+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:33.004 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:33.004 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:33.014 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:33.085 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:33.084+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:33.179 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:33.179 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:33.187 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:33.185+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:33.188 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:33.288 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:33.286+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:33.314 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:33.314 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:33.323 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:33.408 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:33.406+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:33.490 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:33.488+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:33.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '25 of 30' 2024-08-30T16:04:33.504 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:33.505 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:33.514 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:33.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-08-30T16:04:33.599 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:33.597+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:33.679 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:33.679 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:33.688 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:33.730 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:33.728+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:33.791 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:33.790+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:33.853 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:33.854 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:33.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:33.907 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:33.905+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:33.994 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:33.992+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:34.004 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:34.005 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:34.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:34.099 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:34.097+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:34.189 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:34.189 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:34.195 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:34.194+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:34.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:34.297 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:34.295+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:34.368 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:34.368 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:34.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:34.418 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:34.417+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:34.499 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:34.497+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:34.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '26 of 30' 2024-08-30T16:04:34.564 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:34.564 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:34.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:34.582 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-08-30T16:04:34.600 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:34.598+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:34.701 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:34.700+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:34.714 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:34.714 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:34.724 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:34.808 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:34.807+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:34.892 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:34.892 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:34.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:34.902 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:34.900+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:34.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:04:35.030 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:35.029+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:35.072 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:35.072 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:35.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:35.137 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:35.133+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:35.205 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:35.204+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:35.236 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:35.237 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:35.255 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:35.255 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:35.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:35.306 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:35.305+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:35.328 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:35.329 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:35.406 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:35.405+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:35.422 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:35.423 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:35.490 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:35.490 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:35.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:35.490 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:35.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:35.490 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:04:35.490 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:35.490 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 310.11237408400001, 2024-08-30T16:04:35.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:04:35.490 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:35.490 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:35.491 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:35.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:35.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:35.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:35.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:35.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:35.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:35.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:35.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:35.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:35.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:35.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:35.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:35.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:04:35.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:04:35.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:35.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:35.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:35.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:04:35.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:04:35.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:35.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:35.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:35.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:04:35.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:04:35.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:35.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:35.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:35.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:04:35.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:04:35.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:35.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:35.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:35.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:04:35.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:04:35.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:35.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:35.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:35.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:04:35.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:04:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:04:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:04:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:04:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:04:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:35.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:35.495 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:35.495 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:35.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:35.520 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:35.520 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:35.521 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:35.519+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:35.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '27 of 30' 2024-08-30T16:04:35.550 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 7] out_osds: [6, 4, 2, 5] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-08-30T16:04:35.550 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:04:35.550 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-08-30T16:04:35.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-08-30T16:04:35.601 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:35.601 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:35.644 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:35.642+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:35.649 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:35.649 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:35.649 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:35.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:35.707 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:35.707 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:35.707 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:35.706+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:35.808 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:35.807+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:35.828 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:35.829 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:35.829 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:35.829 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:35.829 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:35.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:35.922 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:35.921+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:35.937 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:36.010 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:36.009+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:36.035 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:36.035 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:36.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:36.036 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:36.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.036 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:36.036 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:36.036 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 311.58024033200002, 2024-08-30T16:04:36.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:04:36.036 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:36.036 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:36.036 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:36.036 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:36.036 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:36.037 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:36.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.037 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:36.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:36.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:36.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:36.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:36.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:36.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:04:36.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:04:36.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:36.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:04:36.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:04:36.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:36.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:04:36.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:04:36.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:36.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:04:36.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:04:36.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:36.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:04:36.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:04:36.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:04:36.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:04:36.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:04:36.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:36.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:04:36.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:04:36.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:36.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:04:36.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:04:36.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:04:36.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:04:36.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:04:36.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:36.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:04:36.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:04:36.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:36.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:04:36.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:04:36.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:36.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:04:36.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:04:36.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:36.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:04:36.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:04:36.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.043 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:36.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.043 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:04:36.043 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:36.043 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 310.464373997, 2024-08-30T16:04:36.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:04:36.044 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:36.044 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:36.044 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:36.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:36.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:36.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:36.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:36.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:36.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:04:36.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:36.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:36.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:04:36.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:04:36.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:36.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:04:36.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:04:36.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:36.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:04:36.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:04:36.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:36.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:04:36.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:04:36.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:36.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:04:36.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:04:36.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:36.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:04:36.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:04:36.047 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.047 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:36.047 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.048 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:36.048 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:36.048 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 310.40478249900002, 2024-08-30T16:04:36.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:04:36.048 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:36.048 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:36.048 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:36.048 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:36.048 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:36.048 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:04:36.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.048 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:36.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:36.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:36.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:36.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:36.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:36.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:04:36.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:36.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:36.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:04:36.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:04:36.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:36.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:04:36.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:04:36.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:36.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:04:36.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:04:36.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:36.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:04:36.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:04:36.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:36.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:04:36.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:04:36.053 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.053 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:36.053 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.053 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:04:36.053 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:36.053 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 310.378083812, 2024-08-30T16:04:36.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:04:36.053 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:36.053 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:36.053 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:36.054 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:36.054 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:36.054 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:36.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.054 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:36.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:36.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:36.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:36.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:36.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:36.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:04:36.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:36.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:36.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:04:36.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:04:36.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:36.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:04:36.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:04:36.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:36.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:04:36.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:04:36.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:36.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:04:36.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:04:36.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:36.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 309.51046728400001, 2024-08-30T16:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:36.058 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:36.059 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:36.059 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:36.059 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:36.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.059 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:36.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:36.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:36.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:36.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:36.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:36.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:04:36.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:36.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:36.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:04:36.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:04:36.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:36.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:04:36.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:04:36.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:36.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:04:36.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:04:36.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:36.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:04:36.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:04:36.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:36.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:04:36.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:04:36.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:36.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:04:36.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:04:36.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:36.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:04:36.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:04:36.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:36.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:04:36.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:04:36.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:36.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:04:36.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:04:36.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:36.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:04:36.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:04:36.064 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.064 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:36.064 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.064 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:04:36.065 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:36.065 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 309.351220205, 2024-08-30T16:04:36.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:04:36.065 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:36.065 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:36.065 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:36.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:36.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:36.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:04:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:04:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:36.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:04:36.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:04:36.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:36.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:04:36.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:04:36.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:36.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:04:36.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:04:36.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:36.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:04:36.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:04:36.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:36.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:04:36.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:04:36.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:36.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:04:36.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:04:36.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.069 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:36.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.069 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:36.069 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:36.069 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 309.07504613899999, 2024-08-30T16:04:36.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:04:36.069 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:36.069 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:36.069 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:36.069 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:04:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:04:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:04:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:04:36.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:04:36.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:36.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:04:36.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:04:36.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:36.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:04:36.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:04:36.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:36.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:04:36.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:04:36.073 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.073 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:36.073 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.073 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:36.073 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:36.073 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 308.41675456000002, 2024-08-30T16:04:36.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:04:36.074 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:36.074 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:36.074 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:36.074 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:36.074 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:36.074 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:04:36.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.074 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:36.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:36.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:36.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:36.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:36.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:36.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:04:36.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:36.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:36.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:04:36.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:04:36.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:36.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:04:36.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:04:36.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:36.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:04:36.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:04:36.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:36.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:04:36.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:04:36.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:36.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:04:36.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:04:36.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:36.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:04:36.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:04:36.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:36.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:04:36.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:04:36.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:36.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:04:36.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:04:36.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:36.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:04:36.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:04:36.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:36.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:04:36.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:04:36.079 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.079 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:36.080 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.080 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:04:36.080 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:36.080 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 308.32084127600001, 2024-08-30T16:04:36.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:04:36.080 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:36.081 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:36.081 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:36.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:36.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:36.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:36.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:36.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:36.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:04:36.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:36.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:36.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:04:36.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:04:36.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:36.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:04:36.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:04:36.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:36.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:04:36.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:04:36.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:36.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:04:36.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:04:36.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:36.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:04:36.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:04:36.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:36.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:04:36.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:04:36.084 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.084 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:36.084 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.085 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:36.085 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:36.085 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 249.06948076500001, 2024-08-30T16:04:36.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:04:36.085 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:36.085 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:36.085 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:36.085 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:36.085 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:36.085 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:36.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:04:36.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:04:36.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:36.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:04:36.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:04:36.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:36.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:04:36.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:04:36.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:36.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:04:36.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:04:36.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:36.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:04:36.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:04:36.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:36.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:04:36.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:04:36.089 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.089 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:36.089 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.089 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:04:36.089 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:36.089 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 189.06037607799999, 2024-08-30T16:04:36.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:04:36.089 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:36.089 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:36.089 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:36.090 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:36.090 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:36.090 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:36.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.090 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:36.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:36.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:36.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:36.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:36.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:36.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:04:36.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:04:36.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:36.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:04:36.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:04:36.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:36.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:04:36.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:04:36.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:36.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:04:36.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:04:36.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:36.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:04:36.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:04:36.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:36.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:04:36.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:04:36.093 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.093 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:36.093 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.093 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:11 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-08-30T16:04:36.093 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:27.021982+0000", 2024-08-30T16:04:36.094 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 8.9849928030000008, 2024-08-30T16:04:36.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0095695959999999997, 2024-08-30T16:04:36.094 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:36.094 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:36.094 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:36.094 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:36.094 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:36.094 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 11 2024-08-30T16:04:36.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.094 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:36.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.021982+0000", 2024-08-30T16:04:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.021982+0000", 2024-08-30T16:04:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.021955+0000", 2024-08-30T16:04:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999728 2024-08-30T16:04:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.022110+0000", 2024-08-30T16:04:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015461899999999999 2024-08-30T16:04:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.022347+0000", 2024-08-30T16:04:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023690199999999999 2024-08-30T16:04:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.022532+0000", 2024-08-30T16:04:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000185339 2024-08-30T16:04:36.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:36.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.028853+0000", 2024-08-30T16:04:36.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063211270000000002 2024-08-30T16:04:36.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:36.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:36.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:36.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.031552+0000", 2024-08-30T16:04:36.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026985770000000002 2024-08-30T16:04:36.097 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.097 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:36.098 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.098 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:36.098 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:36.098 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:36.098 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:36.099 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:36.099 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:36.100 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:36.100 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:36.142 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:36.140+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:36.169 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T16:04:36.165+0000 1008f640 -1 mon.a@0(leader).osd e94 definitely_dead 0 2024-08-30T16:04:36.213 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:36.211+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:36.226 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:36.227 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:36.227 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:36.227 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:36.235 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:36.314 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:36.313+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:36.359 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:36.359 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:36.359 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:36.359 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:36.359 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:36.368 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:36.415 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:36.413+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:36.505 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:36.506 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:36.506 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:36.506 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:04:36.506 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:36.515 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:36.514+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:36.518 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:36.528 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '28 of 30' 2024-08-30T16:04:36.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-08-30T16:04:36.605 INFO:teuthology.orchestra.run.smithi050.stderr:marked in osd.4. 2024-08-30T16:04:36.622 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-08-30T16:04:36.622 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:36.621+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:36.644 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:36.644 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:36.644 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:36.644 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:36.654 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:36.767 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:36.765+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:36.789 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:36.789 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:36.789 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:36.790 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:36.790 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:36.800 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:36.893 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:36.891+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:36.994 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:36.993+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:37.070 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:37.071 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:37.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:37.071 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:37.071 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.071 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:37.071 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:37.071 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 326.206585722, 2024-08-30T16:04:37.071 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:04:37.071 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.071 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.072 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.072 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:04:37.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.072 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.072 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:37.072 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.072 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.072 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.072 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:04:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:04:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:04:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:04:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:04:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:04:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:04:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:04:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:04:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:04:37.074 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:04:37.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:04:37.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:04:37.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:04:37.075 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:04:37.075 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.075 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.075 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.075 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:04:37.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:04:37.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:04:37.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:04:37.076 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.076 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.076 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.076 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:04:37.076 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:04:37.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.077 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.077 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:04:37.077 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:04:37.077 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.077 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.077 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.077 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.077 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.077 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:37.078 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:37.078 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 325.29016970200001, 2024-08-30T16:04:37.078 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:04:37.078 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.078 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.078 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.078 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.078 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.078 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:04:37.078 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.078 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.078 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:37.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.079 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.079 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.079 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:37.079 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.079 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:04:37.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:37.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:04:37.080 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:04:37.080 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.080 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.080 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.080 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:04:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:04:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:04:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:04:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:04:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:04:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.081 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:04:37.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:04:37.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:04:37.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:04:37.082 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.082 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.082 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.082 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:04:37.082 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:04:37.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.083 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.083 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:04:37.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:04:37.083 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.083 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.083 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.083 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.083 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.083 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:37.083 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:37.083 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 325.28846514899999, 2024-08-30T16:04:37.083 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:04:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:04:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.084 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:37.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:04:37.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:04:37.085 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.085 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.085 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.085 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:04:37.085 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:04:37.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:04:37.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:04:37.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:04:37.086 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:04:37.086 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.086 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.086 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.086 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.087 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:04:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:04:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:04:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:04:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.088 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 324.86901134200002, 2024-08-30T16:04:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:04:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:04:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.089 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:04:37.090 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:04:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:04:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:04:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:04:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:04:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.091 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:04:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:04:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:04:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:04:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:04:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:04:37.092 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:04:37.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:04:37.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:04:37.093 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:04:37.093 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.093 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.093 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.093 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:04:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:04:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 324.85212316799999, 2024-08-30T16:04:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:04:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.094 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.095 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.095 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:04:37.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.095 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:37.095 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.095 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.095 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.095 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.095 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:37.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:04:37.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:04:37.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.096 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.096 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:04:37.096 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:04:37.096 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.096 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:04:37.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:04:37.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:04:37.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:04:37.097 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.097 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.097 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.097 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:04:37.097 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:04:37.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:04:37.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:04:37.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:04:37.098 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:04:37.098 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.098 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.098 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.098 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:04:37.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:04:37.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.099 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.099 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:04:37.099 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:04:37.099 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.099 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.099 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.099 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.099 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.099 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.099 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:37.099 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 324.83912309900001, 2024-08-30T16:04:37.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:04:37.100 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.100 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.100 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.100 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:04:37.100 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.100 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.100 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.100 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.100 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:37.100 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:37.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.101 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:04:37.101 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:04:37.101 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.101 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.101 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:04:37.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:04:37.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:04:37.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:04:37.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.102 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.102 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.102 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:04:37.102 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:04:37.102 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:04:37.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:04:37.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:04:37.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:04:37.103 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.103 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.103 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.103 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:04:37.103 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:04:37.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:04:37.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:04:37.104 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.104 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.104 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.104 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:04:37.104 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:04:37.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.104 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.104 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 324.83663080700001, 2024-08-30T16:04:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:04:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:04:37.105 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.106 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:04:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:04:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:04:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:04:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:04:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.107 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:04:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:04:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:04:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:04:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:04:37.108 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:04:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:04:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:04:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:04:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:04:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.109 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:04:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:04:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 324.835873359, 2024-08-30T16:04:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:04:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.110 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:04:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.111 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:04:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:04:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:04:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:04:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.112 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:04:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:04:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:04:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:04:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:04:37.113 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:04:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:04:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:04:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:04:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:04:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.114 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:04:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:04:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:04:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:04:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.115 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 324.83514090900002, 2024-08-30T16:04:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:04:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:04:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.116 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:04:37.117 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:04:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:04:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:04:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:04:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.118 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:04:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:04:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:04:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:04:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:04:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:04:37.119 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:04:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:04:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:04:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:04:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.120 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:04:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:04:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 324.83260210999998, 2024-08-30T16:04:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:04:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.121 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:04:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.122 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:04:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:04:37.123 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:04:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:04:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:04:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:04:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:04:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.124 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:04:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:04:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:04:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:04:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:04:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:04:37.125 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:04:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:04:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:04:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:04:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.126 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 324.83185410900001, 2024-08-30T16:04:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:04:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:04:37.127 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.128 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:04:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:04:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:04:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:04:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:04:37.129 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:04:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:04:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:04:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:04:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:04:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.130 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:04:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:04:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:04:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:04:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:04:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:04:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.131 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:04:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:04:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 324.05973763700001, 2024-08-30T16:04:37.132 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:04:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:04:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.133 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:04:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.134 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:04:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:04:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:04:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:04:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:04:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:04:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.135 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:04:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:04:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:04:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:04:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.136 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:04:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:04:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:04:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:04:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:04:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:04:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.137 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 324.05768049099999, 2024-08-30T16:04:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:04:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:04:37.138 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.139 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:04:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:04:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:04:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:04:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:04:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.140 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:04:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:04:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:04:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:04:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:04:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:04:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:04:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:04:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:04:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:04:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.142 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:04:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:04:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 323.79473599099998, 2024-08-30T16:04:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:04:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.143 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:04:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:37.144 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:04:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:04:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:04:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:04:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.145 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:04:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:04:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:04:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:04:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.146 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:04:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:04:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:04:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:04:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:04:37.147 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:04:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:04:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:04:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:04:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:04:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.148 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 323.79364448799998, 2024-08-30T16:04:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:04:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:04:37.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:04:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:04:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:04:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:04:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:04:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:04:37.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:04:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:04:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:04:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:04:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:04:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.153 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:04:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:04:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:04:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:04:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:04:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:04:37.154 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 323.79289490299999, 2024-08-30T16:04:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:04:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.155 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:04:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:04:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:04:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:04:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:04:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:04:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:04:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:04:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:04:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:04:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:04:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:04:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:04:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:04:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:04:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:04:37.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:04:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:04:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 323.759163371, 2024-08-30T16:04:37.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:04:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:04:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:04:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:04:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:04:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:04:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:04:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:04:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:04:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:04:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:04:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:04:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:04:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:04:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:04:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:04:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:04:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:04:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:04:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:04:37.165 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 323.75863736299999, 2024-08-30T16:04:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:04:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.166 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:04:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:04:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:04:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:04:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:04:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:04:37.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:04:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:04:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:04:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:04:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:04:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:04:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:04:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:04:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:04:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:04:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:04:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:04:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 323.75818179999999, 2024-08-30T16:04:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:04:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.171 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:04:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:04:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:04:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:04:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:04:37.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:04:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:04:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:04:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:04:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:04:37.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:04:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:04:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:04:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:04:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:04:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:04:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:04:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:04:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:04:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:04:37.176 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 319.08766648400001, 2024-08-30T16:04:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:04:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:04:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:37.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:04:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:04:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:04:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:04:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:04:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:04:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:04:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.179 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:04:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:04:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:04:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:04:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:04:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:04:37.180 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:04:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:04:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:04:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:04:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.181 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.182 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:37.183 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:37.184 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:37.182+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:37.284 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:37.282+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:37.349 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:37.349 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:37.349 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:37.358 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:37.384 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:37.383+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:37.481 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:37.481 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:37.482 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:37.482 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:37.482 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:37.485 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:37.484+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:37.491 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:37.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '29 of 30' 2024-08-30T16:04:37.603 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:37.602+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:37.616 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-08-30T16:04:37.636 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:37.636 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 309.24575461799998, 2024-08-30T16:04:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:04:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:37.637 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:04:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:04:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:04:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:04:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:04:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:04:37.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:04:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:04:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:04:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:04:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:04:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:04:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:04:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:04:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 189.132758502, 2024-08-30T16:04:37.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:04:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:04:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:37.642 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:04:37.643 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:04:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:04:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:04:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:04:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.644 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:04:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:04:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:04:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:04:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:04:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:04:37.645 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-08-30T16:04:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 69.124339130999999, 2024-08-30T16:04:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037103309999999999, 2024-08-30T16:04:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:37.646 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 10 2024-08-30T16:04:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.647 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496549+0000", 2024-08-30T16:04:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496682+0000", 2024-08-30T16:04:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013371699999999999 2024-08-30T16:04:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.648 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496889+0000", 2024-08-30T16:04:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00020673600000000001 2024-08-30T16:04:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.497052+0000", 2024-08-30T16:04:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000162578 2024-08-30T16:04:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.498833+0000", 2024-08-30T16:04:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017810530000000001 2024-08-30T16:04:37.649 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.500282+0000", 2024-08-30T16:04:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001449437 2024-08-30T16:04:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(2.3 83/82 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 25'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@25'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.743295100000001, 2024-08-30T16:04:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088711689999999999, 2024-08-30T16:04:37.650 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.651 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877588+0000", 2024-08-30T16:04:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999723 2024-08-30T16:04:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.652 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877978+0000", 2024-08-30T16:04:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038947500000000002 2024-08-30T16:04:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.878456+0000", 2024-08-30T16:04:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047795699999999999 2024-08-30T16:04:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.878946+0000", 2024-08-30T16:04:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000490235 2024-08-30T16:04:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.653 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.880512+0000", 2024-08-30T16:04:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001565484 2024-08-30T16:04:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.880749+0000", 2024-08-30T16:04:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000237712 2024-08-30T16:04:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.886487+0000", 2024-08-30T16:04:37.654 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057377349999999999 2024-08-30T16:04:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(2.3 83/82 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-08-30T16:04:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.595887789999999, 2024-08-30T16:04:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062707819999999999, 2024-08-30T16:04:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:37.655 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.656 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025012+0000", 2024-08-30T16:04:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:04:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025103+0000", 2024-08-30T16:04:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0877000000000004e-05 2024-08-30T16:04:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025231+0000", 2024-08-30T16:04:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012737499999999999 2024-08-30T16:04:37.657 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025355+0000", 2024-08-30T16:04:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124194 2024-08-30T16:04:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.026750+0000", 2024-08-30T16:04:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001394922 2024-08-30T16:04:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.658 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.031294+0000", 2024-08-30T16:04:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045441270000000002 2024-08-30T16:04:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.0 91/90 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:26.829025+0000", 2024-08-30T16:04:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.791885702, 2024-08-30T16:04:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.060649472000000003, 2024-08-30T16:04:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.659 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.829025+0000", 2024-08-30T16:04:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.829025+0000", 2024-08-30T16:04:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.660 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.828966+0000", 2024-08-30T16:04:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999404 2024-08-30T16:04:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.830358+0000", 2024-08-30T16:04:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013920580000000001 2024-08-30T16:04:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.830913+0000", 2024-08-30T16:04:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00055505300000000001 2024-08-30T16:04:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.661 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.831281+0000", 2024-08-30T16:04:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000368193 2024-08-30T16:04:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.832628+0000", 2024-08-30T16:04:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001346863 2024-08-30T16:04:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.662 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.833344+0000", 2024-08-30T16:04:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071610100000000002 2024-08-30T16:04:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.889675+0000", 2024-08-30T16:04:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056330587000000001 2024-08-30T16:04:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.0 91/90 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:27.013394+0000", 2024-08-30T16:04:37.663 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 10.607517223, 2024-08-30T16:04:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.021968463000000001, 2024-08-30T16:04:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.013394+0000", 2024-08-30T16:04:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.013394+0000", 2024-08-30T16:04:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:37.664 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:37.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.013362+0000", 2024-08-30T16:04:37.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999681 2024-08-30T16:04:37.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:37.665 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.017776+0000", 2024-08-30T16:04:37.665 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044143150000000003 2024-08-30T16:04:37.665 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.665 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.665 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:37.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.018020+0000", 2024-08-30T16:04:37.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000244076 2024-08-30T16:04:37.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:37.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.018340+0000", 2024-08-30T16:04:37.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00032027100000000001 2024-08-30T16:04:37.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.666 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:37.666 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.019618+0000", 2024-08-30T16:04:37.666 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012770640000000001 2024-08-30T16:04:37.666 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.666 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:37.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.019851+0000", 2024-08-30T16:04:37.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023378 2024-08-30T16:04:37.667 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:37.667 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:37.667 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:37.667 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.035362+0000", 2024-08-30T16:04:37.667 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015510896999999999 2024-08-30T16:04:37.667 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.667 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.667 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.667 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:37.667 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:37.667 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:37.669 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:37.711 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:37.709+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:37.788 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:37.787+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:37.808 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:37.808 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:37.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:37.902 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:37.901+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:37.990 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:37.988+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:37.992 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:37.992 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:37.992 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:37.992 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:37.992 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:38.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:38.090 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:38.089+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:38.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:04:38.191 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:38.190+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:38.292 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:38.291+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:38.314 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(2.3 83/82 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 25'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@25'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.414357915, 2024-08-30T16:04:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.10442118, 2024-08-30T16:04:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895800+0000", 2024-08-30T16:04:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999809 2024-08-30T16:04:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.921337+0000", 2024-08-30T16:04:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.02553677 2024-08-30T16:04:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.926424+0000", 2024-08-30T16:04:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050871340000000001 2024-08-30T16:04:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.935234+0000", 2024-08-30T16:04:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088098540000000006 2024-08-30T16:04:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.944321+0000", 2024-08-30T16:04:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0090873329999999995 2024-08-30T16:04:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.949613+0000", 2024-08-30T16:04:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052919710000000003 2024-08-30T16:04:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:08.000240+0000", 2024-08-30T16:04:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.050627411999999997 2024-08-30T16:04:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:38.320 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:38.320 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:38.324 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:38.376 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:38.377 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:38.389 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:00.994 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:38.390 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:00.995 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:38.393 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:38.392+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:38.419 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:01.025 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:38.438 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:59.974 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:38.439 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:00:59.976 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:38.465 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:00.001 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:38.495 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:38.493+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:38.501 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:38.501 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:38.538 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:38.538 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:38.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:38.623 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:38.623 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:38.624 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:38.623+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:38.696 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:38.694+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:38.715 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:38.715 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:38.720 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:38.720 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:38.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:38.736 INFO:tasks.mon_recovery:adding mon c back in 2024-08-30T16:04:38.736 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-30T16:04:38.737 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-08-30T16:04:38.780 INFO:tasks.ceph.mon.c:Started 2024-08-30T16:04:38.780 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-08-30T16:04:38.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:04:38.798 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:38.798 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:38.799 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:38.797+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:38.897 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:38.897 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:38.899 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:38.898+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:38.900 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:38.914 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:38.914 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:38.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:38.984 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:38.984 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:38.988 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:38.988 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:39.029 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:39.028+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:39.094 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:39.094 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:39.101 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:39.100+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:39.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:39.202 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:39.201+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:39.274 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:39.274 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:39.283 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:39.304 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:39.302+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:39.404 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:39.403+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:39.428 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":33,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:04:39.455 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:39.455 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:39.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:39.505 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:39.504+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:39.606 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:39.605+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:39.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-08-30T16:04:39.651 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:39.651 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:39.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:39.745 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:39.744+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:39.808 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:39.807+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:39.836 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:39.836 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:39.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:39.909 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:39.908+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:40.011 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:40.010+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:40.025 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:40.026 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:40.038 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:40.112 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:40.110+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:40.163 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:40.163 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:40.173 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:40.213 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:40.211+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:40.313 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:40.313+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:40.337 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:40.337 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:40.347 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:40.432 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:40.431+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:40.511 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:40.512 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:40.515 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:40.514+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:40.522 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:40.617 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:40.615+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:40.687 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:40.687 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:40.696 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:40.741 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:40.740+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:40.773 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-08-30T16:04:40.818 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:40.817+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:40.825 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:40.825 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:40.834 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:40.919 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:40.917+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:40.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:04:41.001 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:41.001 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:41.010 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:41.033 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:41.032+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:41.128 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:41.128+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:41.140 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:41.140 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:41.149 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:41.221 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:41.219+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:41.234 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:41.234 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:41.312 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:41.312 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:41.321 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:41.322 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:41.321+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:41.342 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:41.423 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:41.422+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:41.432 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:41.432 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:41.484 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:41.484 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:41.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:41.522 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:41.522 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:41.523 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:41.522+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:41.604 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:41.604 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:41.622 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:41.622 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:41.622 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:41.622 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:41.623 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 7, 4] out_osds: [6, 2, 5] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-08-30T16:04:41.623 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:04:41.623 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-30T16:04:41.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-08-30T16:04:41.624 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:41.623+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:41.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:41.690 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:41.691 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:41.692 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:41.726 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:41.725+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:41.792 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:41.796 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:41.796 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:41.807 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:41.807 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:41.807 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:41.807 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:41.807 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:41.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:41.826 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:41.825+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:41.927 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:41.926+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:41.989 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:41.989 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:41.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:41.989 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:41.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:41.990 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(2.3 83/82 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 25'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@25'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:41.990 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:41.990 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 34.089990233999998, 2024-08-30T16:04:41.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.10442118, 2024-08-30T16:04:41.990 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:41.990 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:41.990 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:41.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:41.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:41.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:41.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:41.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:41.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:41.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:41.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:41.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:41.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:41.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:41.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:41.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895800+0000", 2024-08-30T16:04:41.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999809 2024-08-30T16:04:41.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:41.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:41.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:41.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.921337+0000", 2024-08-30T16:04:41.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.02553677 2024-08-30T16:04:41.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:41.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:41.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:41.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.926424+0000", 2024-08-30T16:04:41.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050871340000000001 2024-08-30T16:04:41.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:41.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:41.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:41.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.935234+0000", 2024-08-30T16:04:41.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088098540000000006 2024-08-30T16:04:41.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:41.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:41.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:41.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.944321+0000", 2024-08-30T16:04:41.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0090873329999999995 2024-08-30T16:04:41.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:41.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:41.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:41.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.949613+0000", 2024-08-30T16:04:41.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052919710000000003 2024-08-30T16:04:41.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:41.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:41.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:41.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:08.000240+0000", 2024-08-30T16:04:41.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.050627411999999997 2024-08-30T16:04:41.993 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:41.993 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:41.994 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:41.994 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:41.994 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:41.994 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:41.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:42.041 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:42.039+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:42.129 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:42.127+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:42.176 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:42.176 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:42.177 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:42.177 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:42.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:42.229 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:42.228+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:42.330 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:42.329+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:42.339 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:04:42.339 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:42.339 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 97, 2024-08-30T16:04:42.339 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "9d72e2e2-0554-42af-8e62-9b1a3a804112", 2024-08-30T16:04:42.339 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-08-30T15:58:20.129410+0000", 2024-08-30T16:04:42.339 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-08-30T16:04:40.585454+0000", 2024-08-30T16:04:42.339 INFO:teuthology.orchestra.run.smithi050.stdout: "last_up_change": "2024-08-30T16:03:35.238525+0000", 2024-08-30T16:04:42.339 INFO:teuthology.orchestra.run.smithi050.stdout: "last_in_change": "2024-08-30T16:04:36.440788+0000", 2024-08-30T16:04:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T16:04:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_num": 5804032, 2024-08-30T16:04:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_set": [ 2024-08-30T16:04:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "nodeep-scrub", 2024-08-30T16:04:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "pglog_hardlimit", 2024-08-30T16:04:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "purged_snapdirs", 2024-08-30T16:04:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_deletes", 2024-08-30T16:04:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "sortbitwise" 2024-08-30T16:04:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:04:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_version": 7, 2024-08-30T16:04:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T16:04:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T16:04:42.340 INFO:teuthology.orchestra.run.smithi050.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T16:04:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_snapshot": "", 2024-08-30T16:04:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_max": 2, 2024-08-30T16:04:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd": 8, 2024-08-30T16:04:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "require_min_compat_client": "luminous", 2024-08-30T16:04:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "min_compat_client": "luminous", 2024-08-30T16:04:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "require_osd_release": "reef", 2024-08-30T16:04:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_crimson": false, 2024-08-30T16:04:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "pools": [ 2024-08-30T16:04:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 1, 2024-08-30T16:04:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".mgr", 2024-08-30T16:04:42.341 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-08-30T15:59:08.355540+0000", 2024-08-30T16:04:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-08-30T16:04:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-08-30T16:04:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-08-30T16:04:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 2, 2024-08-30T16:04:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 1, 2024-08-30T16:04:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-08-30T16:04:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-08-30T16:04:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-08-30T16:04:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T16:04:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T16:04:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-08-30T16:04:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-08-30T16:04:42.342 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 1, 2024-08-30T16:04:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 1, 2024-08-30T16:04:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 1, 2024-08-30T16:04:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 1, 2024-08-30T16:04:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 1, 2024-08-30T16:04:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-08-30T16:04:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-08-30T16:04:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-08-30T16:04:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-08-30T16:04:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-08-30T16:04:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-08-30T16:04:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-08-30T16:04:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.343 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "19", 2024-08-30T16:04:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-08-30T16:04:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T16:04:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T16:04:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-08-30T16:04:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-08-30T16:04:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-08-30T16:04:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-08-30T16:04:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-08-30T16:04:42.344 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-08-30T16:04:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-08-30T16:04:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-08-30T16:04:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-08-30T16:04:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-08-30T16:04:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-08-30T16:04:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-08-30T16:04:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-08-30T16:04:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-08-30T16:04:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-08-30T16:04:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T16:04:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T16:04:42.345 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T16:04:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-08-30T16:04:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-08-30T16:04:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-08-30T16:04:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-08-30T16:04:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-08-30T16:04:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-08-30T16:04:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-08-30T16:04:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-08-30T16:04:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-08-30T16:04:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-08-30T16:04:42.346 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T16:04:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-08-30T16:04:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-08-30T16:04:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-08-30T16:04:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-08-30T16:04:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-08-30T16:04:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-08-30T16:04:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_max": 32, 2024-08-30T16:04:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_min": 1 2024-08-30T16:04:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-08-30T16:04:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": {} 2024-08-30T16:04:42.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-08-30T16:04:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 8, 2024-08-30T16:04:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 8, 2024-08-30T16:04:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.25, 2024-08-30T16:04:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-08-30T16:04:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-08-30T16:04:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-08-30T16:04:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-08-30T16:04:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-08-30T16:04:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-08-30T16:04:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "rbd", 2024-08-30T16:04:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-08-30T15:59:21.696796+0000", 2024-08-30T16:04:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-08-30T16:04:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-08-30T16:04:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-08-30T16:04:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 2, 2024-08-30T16:04:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 1, 2024-08-30T16:04:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-08-30T16:04:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-08-30T16:04:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-08-30T16:04:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T16:04:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T16:04:42.349 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-08-30T16:04:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-08-30T16:04:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 8, 2024-08-30T16:04:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 8, 2024-08-30T16:04:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 8, 2024-08-30T16:04:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 8, 2024-08-30T16:04:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 8, 2024-08-30T16:04:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-08-30T16:04:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-08-30T16:04:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-08-30T16:04:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-08-30T16:04:42.350 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-08-30T16:04:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-08-30T16:04:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-08-30T16:04:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "28", 2024-08-30T16:04:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-08-30T16:04:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T16:04:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T16:04:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-08-30T16:04:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-08-30T16:04:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 3, 2024-08-30T16:04:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 28, 2024-08-30T16:04:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-08-30T16:04:42.351 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-08-30T16:04:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-08-30T16:04:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-08-30T16:04:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-08-30T16:04:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-08-30T16:04:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-08-30T16:04:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-08-30T16:04:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-08-30T16:04:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-08-30T16:04:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-08-30T16:04:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T16:04:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T16:04:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T16:04:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-08-30T16:04:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-08-30T16:04:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-08-30T16:04:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-08-30T16:04:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-08-30T16:04:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-08-30T16:04:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-08-30T16:04:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-08-30T16:04:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-08-30T16:04:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-08-30T16:04:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T16:04:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-08-30T16:04:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-08-30T16:04:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-08-30T16:04:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-08-30T16:04:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-08-30T16:04:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-08-30T16:04:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-08-30T16:04:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd": {} 2024-08-30T16:04:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-08-30T16:04:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 3.9700000286102295, 2024-08-30T16:04:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 3.9700000286102295, 2024-08-30T16:04:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.62999999523162842, 2024-08-30T16:04:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2.5, 2024-08-30T16:04:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2.5, 2024-08-30T16:04:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-08-30T16:04:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-08-30T16:04:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-08-30T16:04:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:04:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-08-30T16:04:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-08-30T16:04:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4", 2024-08-30T16:04:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:04:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:04:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:04:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 15, 2024-08-30T16:04:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 90, 2024-08-30T16:04:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:04:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6812", 2024-08-30T16:04:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:04:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6813", 2024-08-30T16:04:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:04:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.357 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6815", 2024-08-30T16:04:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:04:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6814", 2024-08-30T16:04:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1957399494 2024-08-30T16:04:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6812/1957399494", 2024-08-30T16:04:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6813/1957399494", 2024-08-30T16:04:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6815/1957399494", 2024-08-30T16:04:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6814/1957399494", 2024-08-30T16:04:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:42.359 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:04:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:04:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-08-30T16:04:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "0fe11a1c-4326-4bfd-a2a4-bff14023f301", 2024-08-30T16:04:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:04:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:04:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:04:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:42.360 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 22, 2024-08-30T16:04:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 90, 2024-08-30T16:04:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:04:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6808", 2024-08-30T16:04:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:04:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.362 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:42.362 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.362 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.362 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6809", 2024-08-30T16:04:42.362 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:04:42.362 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.362 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.362 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:42.362 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.362 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.363 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6811", 2024-08-30T16:04:42.363 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:04:42.363 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.363 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.363 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:42.363 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.363 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.363 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6810", 2024-08-30T16:04:42.363 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3208178756 2024-08-30T16:04:42.363 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.363 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.364 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6808/3208178756", 2024-08-30T16:04:42.364 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6809/3208178756", 2024-08-30T16:04:42.364 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6811/3208178756", 2024-08-30T16:04:42.364 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6810/3208178756", 2024-08-30T16:04:42.364 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:42.364 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:04:42.364 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:04:42.364 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.365 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-08-30T16:04:42.365 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "a9d5354b-3600-41ed-80c5-a3439da7aca6", 2024-08-30T16:04:42.365 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:04:42.365 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:04:42.365 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:04:42.365 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:42.365 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:42.365 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:42.365 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 16, 2024-08-30T16:04:42.365 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:04:42.365 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:04:42.366 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:42.366 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:42.366 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.366 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.366 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6804", 2024-08-30T16:04:42.366 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:04:42.366 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.366 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.366 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:42.366 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.366 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6805", 2024-08-30T16:04:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:04:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6807", 2024-08-30T16:04:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:04:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.368 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:42.368 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.368 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.368 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6806", 2024-08-30T16:04:42.368 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 287018800 2024-08-30T16:04:42.368 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.368 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.368 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6804/287018800", 2024-08-30T16:04:42.368 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6805/287018800", 2024-08-30T16:04:42.368 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6807/287018800", 2024-08-30T16:04:42.368 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6806/287018800", 2024-08-30T16:04:42.369 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:42.369 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:04:42.369 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:04:42.369 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.369 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-08-30T16:04:42.369 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "78bcc398-a7e4-4efc-9836-1dfb94407c9c", 2024-08-30T16:04:42.369 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:04:42.369 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:04:42.369 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:04:42.369 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:42.370 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:42.370 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:42.370 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:04:42.370 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 95, 2024-08-30T16:04:42.370 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:04:42.370 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:42.370 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:42.370 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.370 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.370 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6800", 2024-08-30T16:04:42.370 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:04:42.370 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.370 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.371 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:42.371 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.371 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.371 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6801", 2024-08-30T16:04:42.371 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:04:42.371 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.371 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.371 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:42.371 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.371 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.372 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6803", 2024-08-30T16:04:42.372 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:04:42.372 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.372 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.372 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:42.372 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.372 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.372 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6802", 2024-08-30T16:04:42.372 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1140717714 2024-08-30T16:04:42.372 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.372 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.373 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6800/1140717714", 2024-08-30T16:04:42.373 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6801/1140717714", 2024-08-30T16:04:42.373 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6803/1140717714", 2024-08-30T16:04:42.373 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6802/1140717714", 2024-08-30T16:04:42.373 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:42.373 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:04:42.373 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:04:42.373 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.374 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-08-30T16:04:42.374 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "53a1060b-7985-4c2c-820e-64b158056113", 2024-08-30T16:04:42.374 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:04:42.374 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:04:42.374 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:04:42.374 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:42.374 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:42.374 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:42.374 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 20, 2024-08-30T16:04:42.374 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 95, 2024-08-30T16:04:42.374 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:04:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6805", 2024-08-30T16:04:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:04:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.376 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.376 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6806", 2024-08-30T16:04:42.376 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:04:42.376 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.376 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.376 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:42.376 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.376 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.376 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6808", 2024-08-30T16:04:42.376 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:04:42.376 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.377 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.377 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:42.377 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.377 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.377 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6807", 2024-08-30T16:04:42.377 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1468073345 2024-08-30T16:04:42.377 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.377 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.377 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6805/1468073345", 2024-08-30T16:04:42.377 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6806/1468073345", 2024-08-30T16:04:42.377 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6808/1468073345", 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6807/1468073345", 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "853a03f7-227d-4f98-9e5c-c29896820746", 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:42.378 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:42.379 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 13, 2024-08-30T16:04:42.379 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 64, 2024-08-30T16:04:42.379 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:04:42.379 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:42.379 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:42.379 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.379 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.379 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6809", 2024-08-30T16:04:42.379 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:04:42.379 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.379 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.380 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:42.380 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.380 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.380 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6810", 2024-08-30T16:04:42.380 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:04:42.380 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.380 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.380 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:42.380 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.381 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.381 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6812", 2024-08-30T16:04:42.381 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:04:42.381 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.381 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.381 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:42.381 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.381 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.381 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6811", 2024-08-30T16:04:42.381 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3876025616 2024-08-30T16:04:42.381 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.382 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.382 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6809/3876025616", 2024-08-30T16:04:42.382 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6810/3876025616", 2024-08-30T16:04:42.382 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6812/3876025616", 2024-08-30T16:04:42.382 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6811/3876025616", 2024-08-30T16:04:42.382 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:42.382 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:04:42.382 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:04:42.382 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.382 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-08-30T16:04:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "bc978060-bf28-4027-8bba-7d6d8659bfc5", 2024-08-30T16:04:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 0, 2024-08-30T16:04:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-08-30T16:04:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-08-30T16:04:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:04:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 0, 2024-08-30T16:04:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 75, 2024-08-30T16:04:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6801", 2024-08-30T16:04:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:04:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6802", 2024-08-30T16:04:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:04:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.384 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.385 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:42.385 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.385 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.385 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6804", 2024-08-30T16:04:42.385 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:04:42.385 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.385 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.385 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:42.385 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.386 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.386 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6803", 2024-08-30T16:04:42.386 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 128519122 2024-08-30T16:04:42.386 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.386 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.386 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6801/128519122", 2024-08-30T16:04:42.386 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6802/128519122", 2024-08-30T16:04:42.386 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6804/128519122", 2024-08-30T16:04:42.386 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6803/128519122", 2024-08-30T16:04:42.386 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:42.386 INFO:teuthology.orchestra.run.smithi050.stdout: "exists" 2024-08-30T16:04:42.386 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.387 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-08-30T16:04:42.387 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "45df5c6f-b124-46d1-9efb-62b78bc584ed", 2024-08-30T16:04:42.387 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-08-30T16:04:42.387 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-08-30T16:04:42.387 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-08-30T16:04:42.387 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-08-30T16:04:42.387 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-08-30T16:04:42.387 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-08-30T16:04:42.387 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 21, 2024-08-30T16:04:42.387 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 95, 2024-08-30T16:04:42.387 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-08-30T16:04:42.388 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-08-30T16:04:42.388 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-08-30T16:04:42.388 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.388 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.388 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6813", 2024-08-30T16:04:42.388 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:04:42.388 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.388 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.388 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-08-30T16:04:42.388 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.389 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.389 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6814", 2024-08-30T16:04:42.389 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:04:42.389 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.389 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.389 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-08-30T16:04:42.389 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.389 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.389 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6816", 2024-08-30T16:04:42.390 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:04:42.390 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.390 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.390 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-08-30T16:04:42.390 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-08-30T16:04:42.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.390 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-08-30T16:04:42.390 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.94:6815", 2024-08-30T16:04:42.390 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 68721277 2024-08-30T16:04:42.390 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.391 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.391 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.94:6813/68721277", 2024-08-30T16:04:42.391 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.94:6814/68721277", 2024-08-30T16:04:42.391 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.94:6816/68721277", 2024-08-30T16:04:42.391 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.94:6815/68721277", 2024-08-30T16:04:42.391 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-08-30T16:04:42.391 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-08-30T16:04:42.391 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-08-30T16:04:42.391 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.391 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:04:42.392 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_xinfo": [ 2024-08-30T16:04:42.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.392 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-08-30T16:04:42.392 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:42.392 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:42.392 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:42.392 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:42.392 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:04:42.392 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.911895+0000", 2024-08-30T16:04:42.392 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:04:42.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.392 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-08-30T16:04:42.393 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:42.393 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:42.393 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:42.393 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:42.393 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:04:42.393 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:04.786543+0000", 2024-08-30T16:04:42.393 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:04:42.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.393 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-08-30T16:04:42.393 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:42.393 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:42.393 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:42.394 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:42.394 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:04:42.394 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:04.197952+0000", 2024-08-30T16:04:42.394 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:04:42.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.394 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-08-30T16:04:42.394 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:42.394 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:42.394 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:42.394 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:42.394 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:04:42.394 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.680847+0000", 2024-08-30T16:04:42.394 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:04:42.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.395 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-08-30T16:04:42.395 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:42.395 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:42.395 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:42.395 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:42.395 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:04:42.395 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:01.937600+0000", 2024-08-30T16:04:42.395 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:04:42.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.395 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-08-30T16:04:42.396 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:42.396 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:42.396 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:42.396 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:42.396 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:04:42.396 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:02.082761+0000", 2024-08-30T16:04:42.396 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:04:42.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.396 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-08-30T16:04:42.396 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:42.396 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:42.396 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-08-30T16:04:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.401025+0000", 2024-08-30T16:04:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 74 2024-08-30T16:04:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-08-30T16:04:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-08-30T16:04:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-08-30T16:04:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-08-30T16:04:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540138322906710015, 2024-08-30T16:04:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-08-30T16:04:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-08-30T15:59:03.129371+0000", 2024-08-30T16:04:42.397 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-08-30T16:04:42.398 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.398 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:04:42.398 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap": [ 2024-08-30T16:04:42.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.398 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "2.3", 2024-08-30T16:04:42.398 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-08-30T16:04:42.398 INFO:teuthology.orchestra.run.smithi050.stdout: 7, 2024-08-30T16:04:42.398 INFO:teuthology.orchestra.run.smithi050.stdout: 0 2024-08-30T16:04:42.398 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.398 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "2.7", 2024-08-30T16:04:42.398 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-08-30T16:04:42.398 INFO:teuthology.orchestra.run.smithi050.stdout: 7, 2024-08-30T16:04:42.399 INFO:teuthology.orchestra.run.smithi050.stdout: 0 2024-08-30T16:04:42.399 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.399 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.399 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-08-30T16:04:42.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_items": [], 2024-08-30T16:04:42.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_primaries": [], 2024-08-30T16:04:42.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_temp": [], 2024-08-30T16:04:42.399 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_temp": [], 2024-08-30T16:04:42.399 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklist": {}, 2024-08-30T16:04:42.399 INFO:teuthology.orchestra.run.smithi050.stdout: "range_blocklist": {}, 2024-08-30T16:04:42.399 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profiles": { 2024-08-30T16:04:42.399 INFO:teuthology.orchestra.run.smithi050.stdout: "default": { 2024-08-30T16:04:42.399 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-08-30T16:04:42.399 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-08-30T16:04:42.400 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-08-30T16:04:42.400 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-08-30T16:04:42.400 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van" 2024-08-30T16:04:42.400 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.400 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps_queue": [], 2024-08-30T16:04:42.400 INFO:teuthology.orchestra.run.smithi050.stdout: "new_removed_snaps": [], 2024-08-30T16:04:42.400 INFO:teuthology.orchestra.run.smithi050.stdout: "new_purged_snaps": [], 2024-08-30T16:04:42.400 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_node_flags": {}, 2024-08-30T16:04:42.400 INFO:teuthology.orchestra.run.smithi050.stdout: "device_class_flags": {}, 2024-08-30T16:04:42.400 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": { 2024-08-30T16:04:42.400 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_enabled": false, 2024-08-30T16:04:42.401 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_bucket_count": 0, 2024-08-30T16:04:42.401 INFO:teuthology.orchestra.run.smithi050.stdout: "degraded_stretch_mode": 0, 2024-08-30T16:04:42.401 INFO:teuthology.orchestra.run.smithi050.stdout: "recovering_stretch_mode": 0, 2024-08-30T16:04:42.401 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_bucket": 0 2024-08-30T16:04:42.401 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.401 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:42.402 INFO:tasks.thrashosds.thrasher:j is {'epoch': 97, 'fsid': '9d72e2e2-0554-42af-8e62-9b1a3a804112', 'created': '2024-08-30T15:58:20.129410+0000', 'modified': '2024-08-30T16:04:40.585454+0000', 'last_up_change': '2024-08-30T16:03:35.238525+0000', 'last_in_change': '2024-08-30T16:04:36.440788+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T15:59:08.355540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-08-30T15:59:21.696796+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 3.9700000286102295, 'score_stable': 3.9700000286102295, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 90, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6812', 'nonce': 1957399494}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6813', 'nonce': 1957399494}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 1957399494}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6814', 'nonce': 1957399494}]}, 'public_addr': '172.21.15.50:6812/1957399494', 'cluster_addr': '172.21.15.50:6813/1957399494', 'heartbeat_back_addr': '172.21.15.50:6815/1957399494', 'heartbeat_front_addr': '172.21.15.50:6814/1957399494', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0fe11a1c-4326-4bfd-a2a4-bff14023f301', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 90, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6808', 'nonce': 3208178756}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6809', 'nonce': 3208178756}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 3208178756}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6810', 'nonce': 3208178756}]}, 'public_addr': '172.21.15.50:6808/3208178756', 'cluster_addr': '172.21.15.50:6809/3208178756', 'heartbeat_back_addr': '172.21.15.50:6811/3208178756', 'heartbeat_front_addr': '172.21.15.50:6810/3208178756', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a9d5354b-3600-41ed-80c5-a3439da7aca6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6804', 'nonce': 287018800}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6805', 'nonce': 287018800}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 287018800}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6806', 'nonce': 287018800}]}, 'public_addr': '172.21.15.50:6804/287018800', 'cluster_addr': '172.21.15.50:6805/287018800', 'heartbeat_back_addr': '172.21.15.50:6807/287018800', 'heartbeat_front_addr': '172.21.15.50:6806/287018800', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '78bcc398-a7e4-4efc-9836-1dfb94407c9c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6800', 'nonce': 1140717714}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6801', 'nonce': 1140717714}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 1140717714}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.50:6802', 'nonce': 1140717714}]}, 'public_addr': '172.21.15.50:6800/1140717714', 'cluster_addr': '172.21.15.50:6801/1140717714', 'heartbeat_back_addr': '172.21.15.50:6803/1140717714', 'heartbeat_front_addr': '172.21.15.50:6802/1140717714', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '53a1060b-7985-4c2c-820e-64b158056113', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6805', 'nonce': 1468073345}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6806', 'nonce': 1468073345}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6808', 'nonce': 1468073345}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6807', 'nonce': 1468073345}]}, 'public_addr': '172.21.15.94:6805/1468073345', 'cluster_addr': '172.21.15.94:6806/1468073345', 'heartbeat_back_addr': '172.21.15.94:6808/1468073345', 'heartbeat_front_addr': '172.21.15.94:6807/1468073345', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '853a03f7-227d-4f98-9e5c-c29896820746', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 64, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6809', 'nonce': 3876025616}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6810', 'nonce': 3876025616}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6812', 'nonce': 3876025616}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6811', 'nonce': 3876025616}]}, 'public_addr': '172.21.15.94:6809/3876025616', 'cluster_addr': '172.21.15.94:6810/3876025616', 'heartbeat_back_addr': '172.21.15.94:6812/3876025616', 'heartbeat_front_addr': '172.21.15.94:6811/3876025616', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bc978060-bf28-4027-8bba-7d6d8659bfc5', 'up': 0, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 0, 'down_at': 75, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6801', 'nonce': 128519122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6802', 'nonce': 128519122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6804', 'nonce': 128519122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6803', 'nonce': 128519122}]}, 'public_addr': '172.21.15.94:6801/128519122', 'cluster_addr': '172.21.15.94:6802/128519122', 'heartbeat_back_addr': '172.21.15.94:6804/128519122', 'heartbeat_front_addr': '172.21.15.94:6803/128519122', 'state': ['exists']}, {'osd': 7, 'uuid': '45df5c6f-b124-46d1-9efb-62b78bc584ed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6813', 'nonce': 68721277}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6814', 'nonce': 68721277}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6816', 'nonce': 68721277}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.94:6815', 'nonce': 68721277}]}, 'public_addr': '172.21.15.94:6813/68721277', 'cluster_addr': '172.21.15.94:6814/68721277', 'heartbeat_back_addr': '172.21.15.94:6816/68721277', 'heartbeat_front_addr': '172.21.15.94:6815/68721277', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.911895+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:04.786543+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:04.197952+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.680847+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:01.937600+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:02.082761+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.401025+0000', 'dead_epoch': 74}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T15:59:03.129371+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.3', 'osds': [7, 0]}, {'pgid': '2.7', 'osds': [7, 0]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T16:04:42.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:04:42.404 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:42.404 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:42.404 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:42.404 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:42.404 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:42.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:42.431 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:42.430+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:42.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:04:42.532 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:42.530+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:42.585 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:42.585 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:42.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:42.586 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:42.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.586 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:42.586 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:42.586 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 321.30898159399999, 2024-08-30T16:04:42.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:04:42.586 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:42.586 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:42.586 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:42.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:42.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:42.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:42.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:04:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:04:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:42.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:04:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:04:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:04:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:04:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:04:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:04:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:42.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:04:42.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:04:42.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:42.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:04:42.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:04:42.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:42.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:04:42.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:04:42.590 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.590 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.590 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.590 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:42.590 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:42.590 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 321.03605423300002, 2024-08-30T16:04:42.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:04:42.590 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:42.590 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:42.591 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:42.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:42.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:42.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:42.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:42.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:42.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:42.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:04:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:04:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:04:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:04:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:04:42.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:04:42.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:42.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:04:42.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:04:42.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:42.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:04:42.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:04:42.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPull(2.3 e83/82 cost 1049576)", 2024-08-30T16:04:42.594 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:42.595 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 34.751967995000001, 2024-08-30T16:04:42.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051106587000000002, 2024-08-30T16:04:42.595 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:42.595 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:42.595 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:42.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:42.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:42.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:42.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:42.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:42.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:42.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:42.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825962+0000", 2024-08-30T16:04:42.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:42.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:42.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.833441+0000", 2024-08-30T16:04:42.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0074797580000000004 2024-08-30T16:04:42.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:42.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.833651+0000", 2024-08-30T16:04:42.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000209414 2024-08-30T16:04:42.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:42.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.834053+0000", 2024-08-30T16:04:42.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040233199999999998 2024-08-30T16:04:42.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:42.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.835125+0000", 2024-08-30T16:04:42.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010716930000000001 2024-08-30T16:04:42.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:42.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:42.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:42.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.877088+0000", 2024-08-30T16:04:42.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.041962869999999999 2024-08-30T16:04:42.598 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.598 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.598 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.598 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:42.598 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:42.598 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:42.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:42.633 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:42.632+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:42.733 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:42.732+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:42.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-08-30T16:04:42.790 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:42.790 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:42.790 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:42.790 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:42.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:42.834 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:42.834+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:42.935 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:42.934+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:43.028 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:43.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:43.028 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:43.028 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:43.028 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:43.037 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:43.036+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:43.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:43.135 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:04:43.135 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:04:43.137 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:43.136+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-08-30T16:04:41.924600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":795699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":137952,"kb_used_data":2840,"kb_used_omap":7,"kb_used_meta":135032,"kb_avail":471721248,"statfs":{"total":483183820800,"available":483042557952,"internally_reserved":0,"allocated":2908160,"data_stored":2115246,"data_compressed":1558,"data_compressed_allocated":49152,"data_compressed_original":786432,"omap_allocated":7967,"internal_metadata":138272993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":105,"apply_latency_ms":105,"commit_latency_ns":105000000,"apply_latency_ns":105000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000887"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-08-30T16:04:36.799103+0000","last_change":"2024-08-30T16:04:11.379068+0000","last_active":"2024-08-30T16:04:36.799103+0000","last_peered":"2024-08-30T16:04:36.799103+0000","last_clean":"2024-08-30T16:04:36.799103+0000","last_became_active":"2024-08-30T16:04:11.370524+0000","last_became_peered":"2024-08-30T16:04:11.370524+0000","last_unstale":"2024-08-30T16:04:36.799103+0000","last_undegraded":"2024-08-30T16:04:36.799103+0000","last_fullsized":"2024-08-30T16:04:36.799103+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:58:06.441243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2,"reported_epoch":95,"state":"peering","last_fresh":"2024-08-30T16:04:35.139853+0000","last_change":"2024-08-30T16:04:37.499663+0000","last_active":"2024-08-30T16:04:37.499663+0000","last_peered":"2024-08-30T16:04:35.139853+0000","last_clean":"2024-08-30T16:04:35.139853+0000","last_became_active":"2024-08-30T16:04:26.577103+0000","last_became_peered":"2024-08-30T16:04:26.577103+0000","last_unstale":"2024-08-30T16:04:35.139853+0000","last_undegraded":"2024-08-30T16:04:35.139853+0000","last_fullsized":"2024-08-30T16:04:35.139853+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:40:53.077427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-08-30T16:04:37.711140+0000","last_change":"2024-08-30T16:04:37.711140+0000","last_active":"2024-08-30T16:04:37.711140+0000","last_peered":"2024-08-30T16:04:37.711140+0000","last_clean":"2024-08-30T16:04:37.711140+0000","last_became_active":"2024-08-30T16:04:37.703814+0000","last_became_peered":"2024-08-30T16:04:37.703814+0000","last_unstale":"2024-08-30T16:04:37.711140+0000","last_undegraded":"2024-08-30T16:04:37.711140+0000","last_fullsized":"2024-08-30T16:04:37.711140+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T02:31:37.780701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":85,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-08-30T16:04:37.773968+0000","last_change":"2024-08-30T16:04:37.773968+0000","last_active":"2024-08-30T16:04:37.773968+0000","last_peered":"2024-08-30T16:04:37.773968+0000","last_clean":"2024-08-30T16:04:37.773968+0000","last_became_active":"2024-08-30T16:04:37.764408+0000","last_became_peered":"2024-08-30T16:04:37.764408+0000","last_unstale":"2024-08-30T16:04:37.773968+0000","last_undegraded":"2024-08-30T16:04:37.773968+0000","last_fullsized":"2024-08-30T16:04:37.773968+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T00:16:11.648089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":98,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-08-30T16:04:36.726500+0000","last_change":"2024-08-30T16:01:01.435839+0000","last_active":"2024-08-30T16:04:36.726500+0000","last_peered":"2024-08-30T16:04:36.726500+0000","last_clean":"2024-08-30T16:04:36.726500+0000","last_became_active":"2024-08-30T16:01:00.974833+0000","last_became_peered":"2024-08-30T16:01:00.974833+0000","last_unstale":"2024-08-30T16:04:36.726500+0000","last_undegraded":"2024-08-30T16:04:36.726500+0000","last_fullsized":"2024-08-30T16:04:36.726500+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:09:38.575833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":98,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-08-30T16:04:37.756175+0000","last_change":"2024-08-30T16:04:37.756175+0000","last_active":"2024-08-30T16:04:37.756175+0000","last_peered":"2024-08-30T16:04:37.756175+0000","last_clean":"2024-08-30T16:04:37.756175+0000","last_became_active":"2024-08-30T16:04:37.750394+0000","last_became_peered":"2024-08-30T16:04:37.750394+0000","last_unstale":"2024-08-30T16:04:37.756175+0000","last_undegraded":"2024-08-30T16:04:37.756175+0000","last_fullsized":"2024-08-30T16:04:37.756175+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:00:53.987924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":100,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-08-30T16:04:37.685351+0000","last_change":"2024-08-30T16:04:37.685351+0000","last_active":"2024-08-30T16:04:37.685351+0000","last_peered":"2024-08-30T16:04:37.685351+0000","last_clean":"2024-08-30T16:04:37.685351+0000","last_became_active":"2024-08-30T16:04:37.678764+0000","last_became_peered":"2024-08-30T16:04:37.678764+0000","last_unstale":"2024-08-30T16:04:37.685351+0000","last_undegraded":"2024-08-30T16:04:37.685351+0000","last_fullsized":"2024-08-30T16:04:37.685351+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T02:39:43.852577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-08-30T16:04:36.769761+0000","last_change":"2024-08-30T16:04:08.039936+0000","last_active":"2024-08-30T16:04:36.769761+0000","last_peered":"2024-08-30T16:04:36.769761+0000","last_clean":"2024-08-30T16:04:36.769761+0000","last_became_active":"2024-08-30T16:04:07.760364+0000","last_became_peered":"2024-08-30T16:04:07.760364+0000","last_unstale":"2024-08-30T16:04:36.769761+0000","last_undegraded":"2024-08-30T16:04:36.769761+0000","last_fullsized":"2024-08-30T16:04:36.769761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:13:37.016725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":100,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-08-30T16:04:36.611706+0000","last_change":"2024-08-30T16:04:27.064448+0000","last_active":"2024-08-30T16:04:36.611706+0000","last_peered":"2024-08-30T16:04:36.611706+0000","last_clean":"2024-08-30T16:04:36.611706+0000","last_became_active":"2024-08-30T16:04:26.684625+0000","last_became_peered":"2024-08-30T16:04:26.684625+0000","last_unstale":"2024-08-30T16:04:36.611706+0000","last_undegraded":"2024-08-30T16:04:36.611706+0000","last_fullsized":"2024-08-30T16:04:36.611706+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:46:40.994708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280585,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":606208,"data_stored":662639,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":13.491,"5min":12.247,"15min":13.696},"min":{"1min":5.0910000000000002,"5min":4.8049999999999997,"15min":4.8049999999999997},"max":{"1min":45.085000000000001,"5min":57.177999999999997,"15min":57.177999999999997},"last":13.768000000000001},{"interface":"front","average":{"1min":11.670999999999999,"5min":12.109999999999999,"15min":12.93},"min":{"1min":4.9109999999999996,"5min":4.9109999999999996,"15min":4.9109999999999996},"max":{"1min":33.634999999999998,"5min":58.024999999999999,"15min":58.024999999999999},"last":18.539999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":11.814,"5min":11.039,"15min":10.632},"min":{"1min":6.4050000000000002,"5min":5.5780000000000003,"15min":5.5780000000000003},"max":{"1min":45.244,"5min":73.399000000000001,"15min":73.399000000000001},"last":6.4779999999999998},{"interface":"front","average":{"1min":15.143000000000001,"5min":14.789999999999999,"15min":14.981999999999999},"min":{"1min":6.0049999999999999,"5min":6.0049999999999999,"15min":6.0049999999999999},"max":{"1min":43.427,"5min":75.120000000000005,"15min":75.120000000000005},"last":16.306999999999999}]},{"osd":3,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":15.519,"5min":16.143000000000001,"15min":17.157},"min":{"1min":9.7870000000000008,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":29.106999999999999,"5min":66.869,"15min":66.869},"last":20.547000000000001},{"interface":"front","average":{"1min":12.119999999999999,"5min":13.970000000000001,"15min":14.608000000000001},"min":{"1min":7.6520000000000001,"5min":7.6520000000000001,"15min":7.6520000000000001},"max":{"1min":23.492000000000001,"5min":104.38200000000001,"15min":104.38200000000001},"last":14.602}]},{"osd":4,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":14.872999999999999,"5min":15.234,"15min":14.93},"min":{"1min":7.3470000000000004,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":26.141999999999999,"5min":66.340999999999994,"15min":66.340999999999994},"last":22.556999999999999},{"interface":"front","average":{"1min":13.529,"5min":13.217000000000001,"15min":13.115},"min":{"1min":7.0119999999999996,"5min":7.0039999999999996,"15min":7.0039999999999996},"max":{"1min":27.724,"5min":29.818999999999999,"15min":29.818999999999999},"last":23.725999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":17.216999999999999,"5min":16.297000000000001,"15min":16.709},"min":{"1min":8.9320000000000004,"5min":7.8920000000000003,"15min":7.8920000000000003},"max":{"1min":51.021999999999998,"5min":55.784999999999997,"15min":55.784999999999997},"last":19.550999999999998},{"interface":"front","average":{"1min":19.859000000000002,"5min":18.872,"15min":19.481000000000002},"min":{"1min":10.504,"5min":9.5800000000000001,"15min":9.5800000000000001},"max":{"1min":52.845999999999997,"5min":68.004999999999995,"15min":68.004999999999995},"last":17.611999999999998}]},{"osd":6,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":15.708,"5min":16.609000000000002,"15min":17.382000000000001},"min":{"1min":10.180999999999999,"5min":9.1929999999999996,"15min":9.1929999999999996},"max":{"1min":27.353999999999999,"5min":58.201999999999998,"15min":58.201999999999998},"last":18.898},{"interface":"front","average":{"1min":16.324999999999999,"5min":16.748000000000001,"15min":17.001000000000001},"min":{"1min":9.6980000000000004,"5min":8.8390000000000004,"15min":8.8390000000000004},"max":{"1min":26.257999999999999,"5min":37.253,"15min":37.253},"last":12.787000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":17.847999999999999,"5min":18.407,"15min":19.649000000000001},"min":{"1min":11.308,"5min":8.2050000000000001,"15min":8.2050000000000001},"max":{"1min":33.444000000000003,"5min":68.950000000000003,"15min":68.950000000000003},"last":21.782},{"interface":"front","average":{"1min":16.82,"5min":17.565000000000001,"15min":18.876999999999999},"min":{"1min":8.5660000000000007,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":36.186,"5min":59.079000000000001,"15min":59.079000000000001},"last":24.800999999999998}]}]},{"osd":7,"up_from":21,"seq":90194313288,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":662620,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":6.9130000000000003,"5min":8.8819999999999997,"15min":8.9359999999999999},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":11.773999999999999,"5min":30.600999999999999,"15min":30.600999999999999},"last":7.2779999999999996},{"interface":"front","average":{"1min":8.4130000000000003,"5min":9.2750000000000004,"15min":9.173},"min":{"1min":4.2590000000000003,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":19.460999999999999,"5min":28.077000000000002,"15min":28.077000000000002},"last":5.8250000000000002}]},{"osd":1,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":11.005000000000001,"5min":12.624000000000001,"15min":11.432},"min":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"max":{"1min":40.799999999999997,"5min":83.581000000000003,"15min":83.581000000000003},"last":12.641999999999999},{"interface":"front","average":{"1min":11.51,"5min":12.407999999999999,"15min":12.407},"min":{"1min":5.7439999999999998,"5min":5.7439999999999998,"15min":5.7439999999999998},"max":{"1min":38.768999999999998,"5min":81.722999999999999,"15min":81.722999999999999},"last":8.1790000000000003}]},{"osd":2,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":12.596,"5min":13.641999999999999,"15min":13.606},"min":{"1min":5.4850000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"max":{"1min":32.988999999999997,"5min":42.506,"15min":42.506},"last":10.906000000000001},{"interface":"front","average":{"1min":13.048,"5min":14.539,"15min":16.524999999999999},"min":{"1min":8.0050000000000008,"5min":6.9800000000000004,"15min":6.9800000000000004},"max":{"1min":24.622,"5min":43.484999999999999,"15min":43.484999999999999},"last":13.449}]},{"osd":3,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":14.619,"5min":16.202999999999999,"15min":16.465},"min":{"1min":9.0909999999999993,"5min":8.8420000000000005,"15min":8.8420000000000005},"max":{"1min":26.826000000000001,"5min":65.739999999999995,"15min":65.739999999999995},"last":9.5540000000000003},{"interface":"front","average":{"1min":13.359999999999999,"5min":13.337,"15min":12.888999999999999},"min":{"1min":6.4690000000000003,"5min":6.2119999999999997,"15min":6.2119999999999997},"max":{"1min":28.779,"5min":64.980000000000004,"15min":64.980000000000004},"last":6.4409999999999998}]},{"osd":4,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.955,"15min":12.871},"min":{"1min":7.1349999999999998,"5min":7.0339999999999998,"15min":7.0339999999999998},"max":{"1min":41.631,"5min":41.631,"15min":41.631},"last":10.404999999999999},{"interface":"front","average":{"1min":16.166,"5min":16.073,"15min":17.216999999999999},"min":{"1min":8.8729999999999993,"5min":8.8729999999999993,"15min":8.8729999999999993},"max":{"1min":29.516999999999999,"5min":37.393000000000001,"15min":37.393000000000001},"last":11.936}]},{"osd":5,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":11.723000000000001,"5min":11.907,"15min":11.522},"min":{"1min":6.2380000000000004,"5min":6.2309999999999999,"15min":6.2309999999999999},"max":{"1min":33.534999999999997,"5min":33.534999999999997,"15min":33.534999999999997},"last":37.372999999999998},{"interface":"front","average":{"1min":14.628,"5min":15.861000000000001,"15min":16.631},"min":{"1min":8.2870000000000008,"5min":8.2870000000000008,"15min":8.2870000000000008},"max":{"1min":31.853999999999999,"5min":38.618000000000002,"15min":38.618000000000002},"last":38.484000000000002}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.326000000000001,"5min":13.771000000000001,"15min":13.208},"min":{"1min":8.0060000000000002,"5min":6.3230000000000004,"15min":6.3230000000000004},"max":{"1min":19.231000000000002,"5min":42.829999999999998,"15min":42.829999999999998},"last":11.368},{"interface":"front","average":{"1min":15.842000000000001,"5min":17.606000000000002,"15min":18.088999999999999},"min":{"1min":11.712,"5min":10.749000000000001,"15min":10.749000000000001},"max":{"1min":22.695,"5min":44.491999999999997,"15min":44.491999999999997},"last":14.314}]}]},{"osd":6,"up_from":0,"seq":90194313274,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":21,"seq":90194313287,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27584,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344256,"statfs":{"total":96636764160,"available":96608518144,"internally_reserved":0,"allocated":573440,"data_stored":264799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":10.009,"5min":9.9629999999999992,"15min":11.073},"min":{"1min":4.3810000000000002,"5min":4.29,"15min":4.29},"max":{"1min":46.146999999999998,"5min":50.500999999999998,"15min":50.500999999999998},"last":5.4390000000000001},{"interface":"front","average":{"1min":9.9789999999999992,"5min":11.208,"15min":13.714},"min":{"1min":5.3890000000000002,"5min":5.1360000000000001,"15min":5.1360000000000001},"max":{"1min":18.420999999999999,"5min":51.965000000000003,"15min":51.965000000000003},"last":10.157}]},{"osd":1,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":15.244,"5min":15.003,"15min":18.209},"min":{"1min":7.6230000000000002,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":44.862000000000002,"5min":137.56700000000001,"15min":137.56700000000001},"last":12.33},{"interface":"front","average":{"1min":12.853,"5min":13.709,"15min":16.225999999999999},"min":{"1min":5.5720000000000001,"5min":5.0060000000000002,"15min":5.0060000000000002},"max":{"1min":47.530999999999999,"5min":136.321,"15min":136.321},"last":8.3480000000000008}]},{"osd":2,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":12.148,"5min":13.746,"15min":15.852},"min":{"1min":6.3470000000000004,"5min":6.3470000000000004,"15min":6.3470000000000004},"max":{"1min":28.602,"5min":73.831000000000003,"15min":73.831000000000003},"last":7.423},{"interface":"front","average":{"1min":12.590999999999999,"5min":12.757,"15min":15.281000000000001},"min":{"1min":6.2830000000000004,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":26.074999999999999,"5min":71.831999999999994,"15min":71.831999999999994},"last":15.282999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":13.956,"5min":18.161999999999999,"15min":27.286999999999999},"min":{"1min":6.9199999999999999,"5min":6.9199999999999999,"15min":6.9199999999999999},"max":{"1min":27.263999999999999,"5min":342.34500000000003,"15min":342.34500000000003},"last":10.712999999999999},{"interface":"front","average":{"1min":14.484,"5min":16.646999999999998,"15min":19.646999999999998},"min":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"max":{"1min":33.375999999999998,"5min":127.857,"15min":127.857},"last":11.766999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":13.752000000000001,"5min":15.323,"15min":15.843},"min":{"1min":6.5,"5min":6.5,"15min":6.5},"max":{"1min":37.534999999999997,"5min":58.715000000000003,"15min":58.715000000000003},"last":9.4689999999999994},{"interface":"front","average":{"1min":15.122,"5min":17.899000000000001,"15min":18.965},"min":{"1min":8.5310000000000006,"5min":7.2370000000000001,"15min":7.2370000000000001},"max":{"1min":38.450000000000003,"5min":58.212000000000003,"15min":58.212000000000003},"last":12.867000000000001}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.295000000000002,"5min":23.492000000000001,"15min":30.053999999999998},"min":{"1min":11.096,"5min":8.6630000000000003,"15min":8.6630000000000003},"max":{"1min":25.298999999999999,"5min":388.96100000000001,"15min":388.96100000000001},"last":15.696},{"interface":"front","average":{"1min":13.961,"5min":24.18,"15min":32.835999999999999},"min":{"1min":9.4079999999999995,"5min":8.5519999999999996,"15min":8.5519999999999996},"max":{"1min":20.439,"5min":448.69200000000001,"15min":448.69200000000001},"last":13.44}]},{"osd":7,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":22.033000000000001,"5min":20.844999999999999,"15min":25.013000000000002},"min":{"1min":7.3049999999999997,"5min":7.3049999999999997,"15min":7.3049999999999997},"max":{"1min":91.646000000000001,"5min":177.596,"15min":177.596},"last":13.888999999999999},{"interface":"front","average":{"1min":21.826000000000001,"5min":21.407,"15min":26.369},"min":{"1min":7.476,"5min":7.476,"15min":7.476},"max":{"1min":101.158,"5min":231.03800000000001,"15min":231.03800000000001},"last":14.721}]}]},{"osd":4,"up_from":20,"seq":85899345991,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":557056,"data_stored":260408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:04:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7490000000000006,"5min":9.6969999999999992,"15min":11.436999999999999},"min":{"1min":4.5439999999999996,"5min":4.5439999999999996,"15min":4.5439999999999996},"max":{"1min":34.286000000000001,"5min":35.817999999999998,"15min":35.817999999999998},"last":8.9640000000000004},{"interface":"front","average":{"1min":11.119,"5min":10.115,"15min":11.538},"min":{"1min":4.9640000000000004,"5min":4.8470000000000004,"15min":4.8470000000000004},"max":{"1min":35.091999999999999,"5min":35.091999999999999,"15min":35.091999999999999},"last":7.96}]},{"osd":1,"last update":"Fri Aug 30 16:04:28 2024","interfaces":[{"interface":"back","average":{"1min":12.51,"5min":12.608000000000001,"15min":11.776},"min":{"1min":5.6879999999999997,"5min":4.7000000000000002,"15min":4.7000000000000002},"max":{"1min":27.940000000000001,"5min":34.347999999999999,"15min":34.347999999999999},"last":7.1950000000000003},{"interface":"front","average":{"1min":10.170999999999999,"5min":12.247,"15min":12.433999999999999},"min":{"1min":7.3769999999999998,"5min":6.5590000000000002,"15min":6.5590000000000002},"max":{"1min":14.595000000000001,"5min":35.481000000000002,"15min":35.481000000000002},"last":13.686999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:04:23 2024","interfaces":[{"interface":"back","average":{"1min":9.2729999999999997,"5min":9.6370000000000005,"15min":9.9580000000000002},"min":{"1min":6.0949999999999998,"5min":5.4279999999999999,"15min":5.4279999999999999},"max":{"1min":19.388000000000002,"5min":27.201000000000001,"15min":27.201000000000001},"last":6.0659999999999998},{"interface":"front","average":{"1min":11.772,"5min":13.592000000000001,"15min":14.35},"min":{"1min":8.3149999999999995,"5min":8.2390000000000008,"15min":8.2390000000000008},"max":{"1min":23.547000000000001,"5min":30.629000000000001,"15min":30.629000000000001},"last":15.544}]},{"osd":3,"last update":"Fri Aug 30 16:04:23 2024","interfaces":[{"interface":"back","average":{"1min":13.518000000000001,"5min":12.166,"15min":12.31},"min":{"1min":6.7279999999999998,"5min":5.8010000000000002,"15min":5.8010000000000002},"max":{"1min":27.045000000000002,"5min":29.715,"15min":29.715},"last":8.4550000000000001},{"interface":"front","average":{"1min":14.385,"5min":14.616,"15min":15.351000000000001},"min":{"1min":7.8040000000000003,"5min":7.5830000000000002,"15min":7.5830000000000002},"max":{"1min":26.039000000000001,"5min":31.73,"15min":31.73},"last":16.065999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:04:23 2024","interfaces":[{"interface":"back","average":{"1min":11.763999999999999,"5min":12.818,"15min":11.714},"min":{"1min":6.8639999999999999,"5min":6.5010000000000003,"15min":6.5010000000000003},"max":{"1min":26.818999999999999,"5min":42.143000000000001,"15min":42.143000000000001},"last":11.789},{"interface":"front","average":{"1min":13.345000000000001,"5min":13.981,"15min":14.301},"min":{"1min":7.0469999999999997,"5min":6.952,"15min":6.952},"max":{"1min":25.577000000000002,"5min":43.917000000000002,"15min":43.917000000000002},"last":14.537000000000001}]},{"osd":6,"last update":"Fri Aug 30 16:03:23 2024","interfaces":[{"interface":"back","average":{"1min":13.481,"5min":14.797000000000001,"15min":15.696999999999999},"min":{"1min":8.0359999999999996,"5min":6.5289999999999999,"15min":6.5289999999999999},"max":{"1min":20.704000000000001,"5min":37.215000000000003,"15min":37.215000000000003},"last":12.614000000000001},{"interface":"front","average":{"1min":16.030999999999999,"5min":16.199000000000002,"15min":17.021999999999998},"min":{"1min":9.8300000000000001,"5min":8.7870000000000008,"15min":8.7870000000000008},"max":{"1min":24.986999999999998,"5min":37.753999999999998,"15min":37.753999999999998},"last":14.324999999999999}]},{"osd":7,"last update":"Fri Aug 30 16:04:23 2024","interfaces":[{"interface":"back","average":{"1min":14.503,"5min":15.154999999999999,"15min":15.673999999999999},"min":{"1min":8.3460000000000001,"5min":7.7130000000000001,"15min":7.7130000000000001},"max":{"1min":29.631,"5min":34.551000000000002,"15min":34.551000000000002},"last":13.067},{"interface":"front","average":{"1min":15.035,"5min":16.492000000000001,"15min":17.216000000000001},"min":{"1min":5.7110000000000003,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":29.084,"5min":35.701999999999998,"15min":35.701999999999998},"last":10.994}]}]},{"osd":2,"up_from":0,"seq":68719476805,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":15,"seq":64424509509,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":264780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:04:33 2024","interfaces":[{"interface":"back","average":{"1min":10.163,"5min":10.477,"15min":9.9339999999999993},"min":{"1min":4.7489999999999997,"5min":4.7489999999999997,"15min":4.7489999999999997},"max":{"1min":40.735999999999997,"5min":40.735999999999997,"15min":40.735999999999997},"last":7.2110000000000003},{"interface":"front","average":{"1min":12.625999999999999,"5min":11.563000000000001,"15min":11.423},"min":{"1min":4.383,"5min":4.383,"15min":4.383},"max":{"1min":42.387,"5min":42.387,"15min":42.387},"last":8.2230000000000008}]},{"osd":2,"last update":"Fri Aug 30 16:04:14 2024","interfaces":[{"interface":"back","average":{"1min":12.49,"5min":11.603999999999999,"15min":11.340999999999999},"min":{"1min":5.5529999999999999,"5min":3.3370000000000002,"15min":3.3370000000000002},"max":{"1min":30.539999999999999,"5min":46.768999999999998,"15min":46.768999999999998},"last":6.5289999999999999},{"interface":"front","average":{"1min":13.483000000000001,"5min":13.351000000000001,"15min":14.071999999999999},"min":{"1min":6.2350000000000003,"5min":5.702,"15min":5.702},"max":{"1min":29.154,"5min":43.921999999999997,"15min":43.921999999999997},"last":13.192}]},{"osd":3,"last update":"Fri Aug 30 16:04:34 2024","interfaces":[{"interface":"back","average":{"1min":13.814,"5min":16.356000000000002,"15min":22.327999999999999},"min":{"1min":6.6680000000000001,"5min":6.2839999999999998,"15min":6.2839999999999998},"max":{"1min":38.140999999999998,"5min":200.155,"15min":200.155},"last":8.8040000000000003},{"interface":"front","average":{"1min":15.974,"5min":20.137,"15min":27.510999999999999},"min":{"1min":8.4339999999999993,"5min":7.0940000000000003,"15min":7.0940000000000003},"max":{"1min":35.576999999999998,"5min":267.03100000000001,"15min":267.03100000000001},"last":12.561999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:04:34 2024","interfaces":[{"interface":"back","average":{"1min":11.683,"5min":14.086,"15min":15.364000000000001},"min":{"1min":6.0970000000000004,"5min":6.0970000000000004,"15min":6.0970000000000004},"max":{"1min":23.608000000000001,"5min":56.200000000000003,"15min":56.200000000000003},"last":14.891},{"interface":"front","average":{"1min":15.760999999999999,"5min":16.109000000000002,"15min":15.863},"min":{"1min":9.298,"5min":8.609,"15min":8.609},"max":{"1min":24.710000000000001,"5min":48.534999999999997,"15min":48.534999999999997},"last":9.8230000000000004}]},{"osd":5,"last update":"Fri Aug 30 16:04:14 2024","interfaces":[{"interface":"back","average":{"1min":16.681000000000001,"5min":16.052,"15min":16.206},"min":{"1min":5.9199999999999999,"5min":5.6920000000000002,"15min":5.6920000000000002},"max":{"1min":43.308,"5min":43.308,"15min":43.308},"last":11.198},{"interface":"front","average":{"1min":19.315999999999999,"5min":17.890000000000001,"15min":17.920000000000002},"min":{"1min":12.156000000000001,"5min":4.75,"15min":4.75},"max":{"1min":39.359000000000002,"5min":49.396000000000001,"15min":49.396000000000001},"last":10.332000000000001}]},{"osd":6,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":13.808999999999999,"5min":14.292,"15min":14.385999999999999},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":39.460999999999999,"5min":59.328000000000003,"15min":59.328000000000003},"last":23.652999999999999},{"interface":"front","average":{"1min":19.225000000000001,"5min":18.331,"15min":18.533999999999999},"min":{"1min":10.781000000000001,"5min":10.005000000000001,"15min":10.005000000000001},"max":{"1min":46.237000000000002,"5min":58.567,"15min":58.567},"last":38.289000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:04:33 2024","interfaces":[{"interface":"back","average":{"1min":18.983000000000001,"5min":19.995000000000001,"15min":21.375},"min":{"1min":12.195,"5min":9.1180000000000003,"15min":9.1180000000000003},"max":{"1min":34.927999999999997,"5min":59.515999999999998,"15min":59.515999999999998},"last":12.058999999999999},{"interface":"front","average":{"1min":12.894,"5min":14.285,"15min":15.276999999999999},"min":{"1min":7.6379999999999999,"5min":7.1619999999999999,"15min":7.1619999999999999},"max":{"1min":24.966000000000001,"5min":58.807000000000002,"15min":58.807000000000002},"last":14.231999999999999}]}]},{"osd":5,"up_from":0,"seq":55834574917,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:04:43.149 INFO:tasks.thrashosds.thrasher:Setting 2.5 to [2, 5] 2024-08-30T16:04:43.150 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.5', '2', '5'] 2024-08-30T16:04:43.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.5 2 5 2024-08-30T16:04:43.196 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":36,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:04:43.238 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:43.237+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:43.283 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:04:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 317.90598254999998, 2024-08-30T16:04:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:04:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:43.284 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:04:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:04:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:04:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:04:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:04:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:04:43.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:04:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:04:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:04:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:04:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:04:43.287 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:04:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:04:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:04:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:04:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:04:43.288 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.289 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.289 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:43.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:43.339 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:43.337+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:43.440 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:43.438+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:43.458 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:43.458 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:43.459 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:43.459 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:43.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:43.541 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:43.540+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:43.641 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:43.641 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:43.641 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:43.641 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:43.641 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:43.642 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:43.640+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:43.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:43.742 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:43.741+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:43.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:04:43.843 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:43.842+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:43.868 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 319.41361271900001, 2024-08-30T16:04:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:04:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:04:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:04:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:04:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:04:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:04:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:04:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:04:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:04:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:04:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:04:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:04:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:04:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:04:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:04:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:04:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:04:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:04:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:04:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:04:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:04:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:04:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:04:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:04:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:04:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:04:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:04:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:04:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:04:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:04:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 318.29774638399999, 2024-08-30T16:04:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:04:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:04:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:04:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:04:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:04:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:04:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:04:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:04:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:04:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 318.23815488600002, 2024-08-30T16:04:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:04:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:04:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:04:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:04:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:04:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:04:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:04:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:04:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:04:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:04:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:04:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:04:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:04:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:04:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 318.211456199, 2024-08-30T16:04:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:04:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:04:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:04:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:04:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:04:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:04:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:04:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:04:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:04:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:04:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:04:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:04:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 317.34383967100001, 2024-08-30T16:04:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:04:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:04:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:04:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:04:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:04:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:04:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:04:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:04:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:04:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:04:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:04:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:04:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:04:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:04:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:04:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:04:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:04:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:04:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:04:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:04:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:04:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:04:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:04:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 317.184592592, 2024-08-30T16:04:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:04:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:04:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:04:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:04:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:04:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:04:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:04:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:04:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:04:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:04:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:04:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:04:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:04:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:04:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:04:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 316.90841852599999, 2024-08-30T16:04:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:04:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:04:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:04:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:04:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:04:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:04:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:04:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:04:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:04:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:04:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:04:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:04:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 316.25012694700001, 2024-08-30T16:04:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:04:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:04:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:04:43.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:43.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:43.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:04:43.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:04:43.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:43.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:04:43.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:04:43.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:43.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:04:43.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:04:43.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:43.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:04:43.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:04:43.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:43.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:04:43.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:04:43.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:04:43.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:43.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:04:43.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:04:43.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:43.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:04:43.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:04:43.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.908 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.909 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.909 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:04:43.909 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:43.909 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 316.15421366300001, 2024-08-30T16:04:43.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:04:43.909 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:43.909 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:43.909 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:43.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:43.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:43.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:43.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:43.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:43.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:04:43.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:43.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:04:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:04:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:04:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:04:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:04:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:04:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:43.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:04:43.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:04:43.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:43.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:04:43.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:04:43.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:43.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:04:43.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:04:43.913 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.913 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.913 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.913 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:43.913 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:43.913 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 256.90285315199998, 2024-08-30T16:04:43.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:04:43.913 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:43.913 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:43.913 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:43.913 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:43.913 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:43.914 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:43.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.914 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:43.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:43.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:43.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:43.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:43.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:43.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:04:43.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:04:43.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:43.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:04:43.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:04:43.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:43.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:04:43.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:04:43.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:43.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:04:43.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:04:43.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:43.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:04:43.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:04:43.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:43.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:04:43.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:04:43.917 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.917 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.917 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.917 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:04:43.917 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:43.917 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 196.89374846499999, 2024-08-30T16:04:43.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:04:43.917 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:43.917 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:43.918 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:43.918 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:43.918 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:43.918 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:43.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.918 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:43.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:43.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:43.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:43.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:43.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:43.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:04:43.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:04:43.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:43.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:04:43.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:04:43.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:43.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:04:43.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:04:43.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:43.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:04:43.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:04:43.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:43.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:04:43.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:04:43.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:43.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:04:43.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:04:43.921 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.921 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.921 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.921 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:11 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-08-30T16:04:43.921 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:27.021982+0000", 2024-08-30T16:04:43.921 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 16.818365190000002, 2024-08-30T16:04:43.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0095695959999999997, 2024-08-30T16:04:43.921 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:43.922 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:43.922 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:43.922 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:43.922 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:43.922 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 11 2024-08-30T16:04:43.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.922 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:43.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:43.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.021982+0000", 2024-08-30T16:04:43.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:43.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.021982+0000", 2024-08-30T16:04:43.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:43.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:43.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.021955+0000", 2024-08-30T16:04:43.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999728 2024-08-30T16:04:43.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:43.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.022110+0000", 2024-08-30T16:04:43.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015461899999999999 2024-08-30T16:04:43.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:43.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.022347+0000", 2024-08-30T16:04:43.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023690199999999999 2024-08-30T16:04:43.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:43.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.022532+0000", 2024-08-30T16:04:43.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000185339 2024-08-30T16:04:43.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:43.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.028853+0000", 2024-08-30T16:04:43.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063211270000000002 2024-08-30T16:04:43.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:43.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:43.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:43.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.031552+0000", 2024-08-30T16:04:43.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026985770000000002 2024-08-30T16:04:43.925 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.925 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.925 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.925 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:43.925 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:43.925 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:43.926 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:43.971 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:43.970+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:44.035 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:44.035 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:44.045 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:44.043+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:44.050 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:44.050 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:44.050 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:44.050 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:44.059 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:44.126 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:44.145 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:44.144+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:44.217 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:44.217 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:44.224 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:44.225 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:44.225 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:44.225 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:44.225 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:44.234 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:44.275 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:44.274+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:44.311 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:44.311 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:44.347 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:44.346+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:44.433 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:44.433 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:44.434 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:44.434 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:44.444 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:44.447 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:44.446+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:44.513 INFO:teuthology.orchestra.run.smithi050.stderr:set 2.5 pg_upmap mapping to [2,5] 2024-08-30T16:04:44.523 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-08-30T16:04:44.548 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:44.548 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:44.549 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:44.547+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:44.551 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:44.612 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:44.612 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:44.622 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:44.636 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:44.636 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:44.640 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:44.681 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:44.680+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:44.745 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:44.745 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:44.749 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:44.748+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:44.754 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:44.850 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:44.849+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:44.877 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:44.877 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:44.887 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:44.951 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:44.949+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:45.051 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:45.051 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:45.051 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:45.051+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:45.060 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:45.153 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:45.151+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:45.224 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:45.224 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:45.234 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:45.257 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:45.256+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:45.354 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:45.353+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:45.361 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:45.361 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:45.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:45.455 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:45.454+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:45.502 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:45.502 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:45.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:45.556 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:45.555+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:45.641 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:45.641 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:45.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:45.691 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:45.690+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:45.758 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:45.757+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:45.777 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:45.777 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:45.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:45.871 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:45.870+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:45.958 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:45.958 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:45.959 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:45.958+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:45.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:46.061 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:46.059+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:46.136 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:46.136 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:46.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:46.188 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:46.185+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:46.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:04:46.272 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:46.271+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:46.321 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:46.321 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:46.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:46.371 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:46.370+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:46.484 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:46.483+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:46.498 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:46.498 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:46.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:46.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-08-30T16:04:46.566 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:46.564+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:46.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:04:46.685 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:46.684+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:46.694 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:46.694 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:46.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:46.792 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:46.791+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:46.868 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:46.867+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:46.895 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:08.055 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:46.902 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":40,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:04:46.904 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:46.904 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:46.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:47.003 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:47.001+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:47.072 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:47.070+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:47.089 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:47.089 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:47.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:47.144 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:08.304 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:47.172 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:47.170+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:47.230 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:47.230 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:47.277 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:47.272+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:47.300 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:47.301 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:47.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:47.342 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:47.343 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:47.433 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:47.433 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:47.433 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:47.431+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:47.488 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:47.486+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:47.489 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:47.489 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:47.502 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:47.556 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:47.557 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:47.576 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:47.574+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:47.659 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:47.659 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:47.660 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:47.660 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:47.660 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:47.660 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:47.672 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:47.714 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:47.712+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:47.758 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:47.758 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:47.778 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:47.776+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:47.799 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:47.856 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:47.856 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:47.856 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:47.856 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:47.856 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:47.864 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-08-30T16:04:47.866 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:47.900 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:47.900 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:47.904 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:47.909 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:47.906+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:47.980 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:47.978+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:48.039 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:48.039 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:04:48.039 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:48.049 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:48.093 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:48.091+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:48.174 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:48.174 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:48.175 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:48.175 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:48.181 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:48.180+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:48.183 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:48.282 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:48.280+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:48.347 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:48.347 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:48.347 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:48.347 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:48.348 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:48.356 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:48.400 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:48.398+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:48.484 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:48.482+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:48.528 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:48.528 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 337.66642373100001, 2024-08-30T16:04:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:04:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.529 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:04:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:04:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:04:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:04:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:04:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.531 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:04:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:04:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:04:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:04:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:04:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:04:48.532 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:04:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:04:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:04:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:04:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:04:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.533 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:04:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:04:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:04:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:04:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:04:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:04:48.534 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:04:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:04:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 336.75000771100002, 2024-08-30T16:04:48.535 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:04:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:04:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.536 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:04:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:04:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:04:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:04:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:04:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.538 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:04:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:04:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:04:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:04:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:04:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:04:48.539 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:04:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:04:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:04:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:04:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.540 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:04:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:04:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 336.748303158, 2024-08-30T16:04:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:04:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.541 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:04:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:48.542 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:04:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:04:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:04:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:04:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.543 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:04:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:04:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:04:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:04:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.544 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:04:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:04:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:04:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:04:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:04:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:04:48.545 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:04:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:04:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:04:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:04:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.546 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 336.32884935099997, 2024-08-30T16:04:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:04:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:04:48.547 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.548 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:04:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:04:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:04:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:04:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:04:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:04:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.549 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:04:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:04:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:04:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:04:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:04:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:04:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.550 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:04:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:04:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:04:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:04:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:04:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:04:48.551 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 336.311961177, 2024-08-30T16:04:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:04:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.552 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:04:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.553 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:04:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:04:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:04:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:04:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.554 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:04:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:04:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:04:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:04:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:04:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:04:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.555 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:04:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:04:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:04:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:04:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:04:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:04:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.556 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:04:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:04:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 336.29896110800001, 2024-08-30T16:04:48.557 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:04:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:04:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.558 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.559 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:04:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:04:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:04:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:04:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.560 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:04:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:04:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:04:48.561 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:04:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:04:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:04:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:04:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:04:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.562 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:04:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:04:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:04:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:04:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.563 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:04:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:04:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 336.29646881600002, 2024-08-30T16:04:48.564 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:04:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:04:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.565 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:04:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:48.566 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:04:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:04:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:04:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:04:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.567 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:04:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:04:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:04:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:04:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.568 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:04:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:04:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:04:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:04:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.569 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:04:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:04:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:04:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:04:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.570 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 336.29571136800001, 2024-08-30T16:04:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:04:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.571 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:04:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.572 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:04:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:04:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.573 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:04:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:04:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:04:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:04:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:04:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:04:48.574 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:04:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:04:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:04:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:04:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.575 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:04:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:04:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:04:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:04:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:04:48.576 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:04:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:48.577 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 336.29497891800003, 2024-08-30T16:04:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:04:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:04:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.578 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:04:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.579 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:04:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:04:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:04:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:04:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:04:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:04:48.580 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:04:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:04:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:04:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:04:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.581 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:04:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:04:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:04:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:04:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:04:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:04:48.582 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 336.29244011899999, 2024-08-30T16:04:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:04:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.583 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:04:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.584 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:04:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:04:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:04:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.585 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:04:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:04:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:04:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:04:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:04:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:04:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.586 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:04:48.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:04:48.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:04:48.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:04:48.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.587 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.587 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.587 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:04:48.587 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:04:48.587 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.588 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.588 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:04:48.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:04:48.588 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.588 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.588 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.588 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.588 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.588 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.588 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:48.588 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 336.29169211800001, 2024-08-30T16:04:48.588 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:04:48.589 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.589 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.589 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.589 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.589 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.589 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:04:48.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.589 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.589 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.589 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.589 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:48.589 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.589 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:04:48.590 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:04:48.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:04:48.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:04:48.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.591 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.591 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:04:48.591 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:04:48.591 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.591 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:04:48.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:04:48.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:04:48.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:04:48.592 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.592 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.592 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.592 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:04:48.592 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:04:48.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:04:48.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:04:48.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.593 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.593 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.593 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:04:48.593 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:04:48.593 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.593 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.593 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.593 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 335.51957564600002, 2024-08-30T16:04:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:04:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:04:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:48.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:48.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.595 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.595 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.595 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:04:48.595 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:48.595 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:04:48.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:04:48.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:04:48.596 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:04:48.596 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.596 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.596 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.596 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:04:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:04:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:04:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:04:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.597 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:04:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:04:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:04:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:04:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:04:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:04:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.598 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 335.51751849999999, 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.599 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:04:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.600 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:04:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:04:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:04:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.601 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:04:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:04:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:04:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:04:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:04:48.602 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:04:48.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:04:48.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:04:48.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.603 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:04:48.603 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:04:48.603 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.603 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.603 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:04:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:04:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:04:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:04:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.604 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 335.25457399999999, 2024-08-30T16:04:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:04:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:04:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.605 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:04:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:04:48.606 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:04:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:04:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:04:48.607 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:04:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:04:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:04:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:04:48.608 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:04:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:04:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:04:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:04:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:04:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.609 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:04:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:04:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:04:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:04:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.610 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 335.25348249699999, 2024-08-30T16:04:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:04:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:04:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.611 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:04:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:48.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:04:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:04:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:04:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:04:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.613 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:04:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:04:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:04:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:04:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:04:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:04:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.614 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:04:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:04:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:04:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:04:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:04:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:04:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.615 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 335.252732912, 2024-08-30T16:04:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:04:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.616 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:04:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:04:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:04:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:04:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:04:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:04:48.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:04:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:04:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:04:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:04:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:04:48.619 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:04:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:04:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:04:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:04:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:04:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.620 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:04:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:04:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 335.21900138000001, 2024-08-30T16:04:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:04:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.621 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:04:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.622 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:04:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:04:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:04:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:04:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.623 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:04:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:04:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:04:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:04:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:04:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:04:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.624 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:04:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:04:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:04:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:04:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:04:48.625 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:04:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:04:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:04:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 335.218475372, 2024-08-30T16:04:48.626 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:04:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:04:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.627 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:04:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:04:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:04:48.628 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:04:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:04:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:04:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:04:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.629 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:04:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:04:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:04:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:04:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:04:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:04:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.630 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:04:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:04:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:04:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:04:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:48.631 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 335.218019809, 2024-08-30T16:04:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:04:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:04:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:48.632 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:04:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:04:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.633 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:04:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:04:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:04:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:04:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:04:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:04:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:04:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:04:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:04:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:04:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:04:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:04:48.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:04:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:04:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:04:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:04:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:04:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 330.54750449300002, 2024-08-30T16:04:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:04:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:04:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:48.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:04:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:48.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:04:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:04:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:04:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:04:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:04:48.639 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:04:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:04:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:04:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:04:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:04:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.640 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:04:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:04:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:04:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:04:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:04:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:04:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:48.642 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:48.642 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:48.643 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:48.643 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:48.641+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:48.744 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:48.743+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:48.822 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:48.822 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:48.822 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:48.832 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:48.876 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:48.874+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:48.947 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:48.945+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:48.956 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:48.956 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:48.956 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:48.956 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:48.957 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:48.966 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:49.050 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:49.048+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:49.148 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:49.148 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:49.148 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:49.148 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:49.148 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.148 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:49.148 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:49.149 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 320.75801941200001, 2024-08-30T16:04:49.149 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:04:49.149 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:49.149 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:49.149 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:49.149 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:49.149 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:49.149 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:04:49.149 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.149 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:49.149 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.149 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:49.149 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:49.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:49.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:49.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:49.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:49.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.150 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:49.150 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:04:49.150 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:04:49.150 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.150 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:49.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:04:49.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:04:49.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:49.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:04:49.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:04:49.151 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.151 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.151 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:49.151 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:04:49.151 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:04:49.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:49.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:04:49.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:04:49.152 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.152 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.152 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:49.152 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:04:49.152 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:04:49.152 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.152 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:49.153 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.153 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.153 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.153 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:04:49.153 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:49.153 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 200.64502329600001, 2024-08-30T16:04:49.153 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:04:49.153 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:49.153 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:49.153 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:49.153 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:49.153 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:49.154 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:04:49.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.154 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:49.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:49.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:49.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:49.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.154 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.154 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:49.154 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:49.154 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:49.154 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:49.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:04:49.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:49.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:49.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:04:49.155 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:04:49.155 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.155 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.155 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:49.155 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:04:49.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:04:49.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:49.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:04:49.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:04:49.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:49.156 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:04:49.156 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:04:49.156 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.156 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.156 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:04:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:04:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-08-30T16:04:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 80.636603925000003, 2024-08-30T16:04:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037103309999999999, 2024-08-30T16:04:49.157 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:49.158 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:49.158 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:49.158 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:49.158 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:49.158 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 10 2024-08-30T16:04:49.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.158 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:49.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:49.158 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:49.158 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:49.158 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.158 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.158 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:49.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:49.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:49.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:49.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496549+0000", 2024-08-30T16:04:49.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:49.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.159 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.159 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:49.159 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496682+0000", 2024-08-30T16:04:49.159 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013371699999999999 2024-08-30T16:04:49.159 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:49.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496889+0000", 2024-08-30T16:04:49.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00020673600000000001 2024-08-30T16:04:49.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:49.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.497052+0000", 2024-08-30T16:04:49.160 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000162578 2024-08-30T16:04:49.160 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.160 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.160 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:49.160 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.498833+0000", 2024-08-30T16:04:49.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017810530000000001 2024-08-30T16:04:49.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.161 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:49.161 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.500282+0000", 2024-08-30T16:04:49.161 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001449437 2024-08-30T16:04:49.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.161 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:49.161 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.161 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.161 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.161 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(2.3 83/82 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 25'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@25'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:49.161 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:49.162 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.255559894000001, 2024-08-30T16:04:49.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088711689999999999, 2024-08-30T16:04:49.162 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:49.162 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:49.162 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:49.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:49.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:49.162 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:49.162 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.162 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.162 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:49.162 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:49.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:49.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:49.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877588+0000", 2024-08-30T16:04:49.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999723 2024-08-30T16:04:49.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.163 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.163 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:49.163 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877978+0000", 2024-08-30T16:04:49.163 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038947500000000002 2024-08-30T16:04:49.163 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:49.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.878456+0000", 2024-08-30T16:04:49.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047795699999999999 2024-08-30T16:04:49.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:49.164 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.878946+0000", 2024-08-30T16:04:49.164 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000490235 2024-08-30T16:04:49.164 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.164 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:49.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.880512+0000", 2024-08-30T16:04:49.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001565484 2024-08-30T16:04:49.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:49.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.880749+0000", 2024-08-30T16:04:49.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000237712 2024-08-30T16:04:49.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.165 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:49.165 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.886487+0000", 2024-08-30T16:04:49.165 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057377349999999999 2024-08-30T16:04:49.165 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.166 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:49.166 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.166 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(2.3 83/82 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-08-30T16:04:49.166 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:49.166 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 41.108152584000003, 2024-08-30T16:04:49.166 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062707819999999999, 2024-08-30T16:04:49.166 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:49.166 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:49.166 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:49.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.166 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:49.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:49.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:49.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:49.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:49.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:49.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.167 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:49.167 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025012+0000", 2024-08-30T16:04:49.167 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:04:49.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:49.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025103+0000", 2024-08-30T16:04:49.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0877000000000004e-05 2024-08-30T16:04:49.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:49.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025231+0000", 2024-08-30T16:04:49.168 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012737499999999999 2024-08-30T16:04:49.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.168 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:49.168 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025355+0000", 2024-08-30T16:04:49.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124194 2024-08-30T16:04:49.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:49.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.026750+0000", 2024-08-30T16:04:49.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001394922 2024-08-30T16:04:49.169 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.169 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.169 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:49.169 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.031294+0000", 2024-08-30T16:04:49.169 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045441270000000002 2024-08-30T16:04:49.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.169 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:49.169 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.170 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.170 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.0 91/90 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:49.170 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:26.829025+0000", 2024-08-30T16:04:49.170 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.304150495999998, 2024-08-30T16:04:49.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.060649472000000003, 2024-08-30T16:04:49.170 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:49.170 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:49.170 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:49.170 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.170 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:49.170 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.829025+0000", 2024-08-30T16:04:49.170 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:49.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:49.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.829025+0000", 2024-08-30T16:04:49.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:49.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:49.171 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.828966+0000", 2024-08-30T16:04:49.171 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999404 2024-08-30T16:04:49.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.171 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.830358+0000", 2024-08-30T16:04:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013920580000000001 2024-08-30T16:04:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.830913+0000", 2024-08-30T16:04:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00055505300000000001 2024-08-30T16:04:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.831281+0000", 2024-08-30T16:04:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000368193 2024-08-30T16:04:49.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.832628+0000", 2024-08-30T16:04:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001346863 2024-08-30T16:04:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.833344+0000", 2024-08-30T16:04:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071610100000000002 2024-08-30T16:04:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.173 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.889675+0000", 2024-08-30T16:04:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056330587000000001 2024-08-30T16:04:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.0 91/90 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:27.013394+0000", 2024-08-30T16:04:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 22.119782016999999, 2024-08-30T16:04:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.021968463000000001, 2024-08-30T16:04:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:49.174 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.013394+0000", 2024-08-30T16:04:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.013394+0000", 2024-08-30T16:04:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:49.175 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.013362+0000", 2024-08-30T16:04:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999681 2024-08-30T16:04:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.017776+0000", 2024-08-30T16:04:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044143150000000003 2024-08-30T16:04:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.018020+0000", 2024-08-30T16:04:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000244076 2024-08-30T16:04:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.018340+0000", 2024-08-30T16:04:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00032027100000000001 2024-08-30T16:04:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.019618+0000", 2024-08-30T16:04:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012770640000000001 2024-08-30T16:04:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:49.177 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.019851+0000", 2024-08-30T16:04:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023378 2024-08-30T16:04:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.035362+0000", 2024-08-30T16:04:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015510896999999999 2024-08-30T16:04:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:49.178 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:49.178 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:49.179 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:49.147+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:49.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:49.249 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:49.248+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:49.309 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:49.309 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:49.309 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:49.309 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:49.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:49.364 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:49.361+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:49.451 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:49.449+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:49.501 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:49.501 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:49.501 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:49.501 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:49.501 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:49.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:49.528 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 7, 4] out_osds: [6, 2, 5] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-08-30T16:04:49.528 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-08-30T16:04:49.529 INFO:tasks.ceph.osd.6:Restarting daemon 2024-08-30T16:04:49.529 DEBUG:teuthology.orchestra.run.smithi094:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-08-30T16:04:49.531 INFO:tasks.ceph.osd.6:Started 2024-08-30T16:04:49.531 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:04:49.552 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:49.550+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:49.622 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-08-30T16:04:49.624 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T16:04:49.625 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-08-30T16:04:49.652 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:49.651+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:49.681 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(2.3 83/82 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 25'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@25'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 41.781948366000002, 2024-08-30T16:04:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.10442118, 2024-08-30T16:04:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895800+0000", 2024-08-30T16:04:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999809 2024-08-30T16:04:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.921337+0000", 2024-08-30T16:04:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.02553677 2024-08-30T16:04:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:49.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:49.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:49.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.926424+0000", 2024-08-30T16:04:49.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050871340000000001 2024-08-30T16:04:49.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:49.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:49.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:49.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.935234+0000", 2024-08-30T16:04:49.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088098540000000006 2024-08-30T16:04:49.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:49.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:49.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:49.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.944321+0000", 2024-08-30T16:04:49.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0090873329999999995 2024-08-30T16:04:49.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:49.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:49.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:49.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.949613+0000", 2024-08-30T16:04:49.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052919710000000003 2024-08-30T16:04:49.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:49.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:49.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:49.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:08.000240+0000", 2024-08-30T16:04:49.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.050627411999999997 2024-08-30T16:04:49.686 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:49.686 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:49.686 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:49.686 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:49.686 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:49.686 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:49.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:49.754 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:49.752+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:49.829 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:49.829 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:49.829 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:49.829 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:49.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:49.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-08-30T16:04:49.883 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:49.882+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:49.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T16:04:49.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:04:49.971 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:49.969+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:49.999 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:49.999 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:49.999 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:49.999 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:49.999 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:50.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:50.057 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:50.056+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:50.070 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:11.230 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:50.157 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:50.156+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:50.158 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:11.317 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:50.248 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:50.248 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:50.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:50.248 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:50.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.248 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:50.249 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:50.249 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 328.97158248199997, 2024-08-30T16:04:50.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:04:50.249 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:50.249 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:50.249 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:50.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:50.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:50.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:50.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:50.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:50.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:50.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:50.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:04:50.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:04:50.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.251 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:50.251 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:04:50.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:04:50.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.251 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:50.251 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:04:50.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:04:50.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.251 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:50.251 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:04:50.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:04:50.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:50.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:04:50.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:04:50.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:50.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:04:50.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:04:50.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:50.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:04:50.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:04:50.253 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:50.253 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:50.253 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:50.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.253 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:50.253 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 328.698655121, 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:50.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:50.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:04:50.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:04:50.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:50.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:04:50.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:04:50.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:50.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:04:50.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:04:50.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:50.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:04:50.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:04:50.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:04:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:04:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:04:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:04:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:04:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:04:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:50.257 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPull(2.3 e83/82 cost 1049576)", 2024-08-30T16:04:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 42.414568883000001, 2024-08-30T16:04:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051106587000000002, 2024-08-30T16:04:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:50.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825962+0000", 2024-08-30T16:04:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.833441+0000", 2024-08-30T16:04:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0074797580000000004 2024-08-30T16:04:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.833651+0000", 2024-08-30T16:04:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000209414 2024-08-30T16:04:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.834053+0000", 2024-08-30T16:04:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040233199999999998 2024-08-30T16:04:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.835125+0000", 2024-08-30T16:04:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010716930000000001 2024-08-30T16:04:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.877088+0000", 2024-08-30T16:04:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.041962869999999999 2024-08-30T16:04:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:50.261 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:50.261 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:50.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:50.263 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:50.261+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:50.273 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:50.274 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:50.364 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:50.362+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:50.382 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:50.383 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:50.428 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:50.428 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:50.428 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:50.429 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:50.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:50.465 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:50.463+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:50.507 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:11.666 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:50.527 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:50.527 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:50.565 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:50.564+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:50.612 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:50.612 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:50.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:50.651 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:50.651 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:50.667 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:50.665+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:50.732 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":44,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:04:50.737 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:50.738 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:50.762 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:50.763 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:50.767 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:50.766+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:50.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:50.826 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:50.827 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:50.829 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:50.869 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:50.867+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:50.941 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:50.941 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:50.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:50.948 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:50.948 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:50.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:51.001 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:50.999+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:51.070 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:51.069+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:51.130 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:51.130 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:51.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:51.171 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:51.169+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:51.271 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:51.270+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:51.310 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:51.310 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:51.319 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:51.373 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:51.371+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:51.450 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:51.450 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:51.459 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:51.474 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:51.472+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:51.575 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:51.573+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:51.584 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:51.585 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:51.594 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:51.675 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:51.674+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:51.724 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:51.724 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:51.733 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:51.777 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:51.775+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:51.877 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:51.875+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:51.920 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:51.920 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:51.930 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:51.978 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:51.976+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:52.074 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:52.074 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:52.078 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:52.077+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:52.083 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:52.179 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:04:52.178+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:04:52.223 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:52.224 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:52.234 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:52.281 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:52.279+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:52.361 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:52.361 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:52.371 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:52.412 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:52.410+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:52.482 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:52.481+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:52.536 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:52.536 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:52.545 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:52.589 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:52.587+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:52.677 INFO:teuthology.orchestra.run.smithi094.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:52.678 INFO:teuthology.orchestra.run.smithi094.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:52.684 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:52.682+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:52.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T16:04:52.725 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-08-30T16:04:52.785 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:52.783+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:52.834 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:52.834 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:52.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-08-30T16:04:52.886 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:52.884+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:52.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T16:04:52.970 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-08-30T16:04:52.970 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T16:04:52.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-08-30T16:04:52.987 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:52.985+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:53.088 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:53.086+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:53.189 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:04:53.188+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:04:53.199 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-08-30T16:04:53.199 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T16:04:53.204 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:53.205 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:53.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:53.205 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:53.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.205 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(2.3 83/82 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 25'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@25'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:53.206 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:53.206 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 45.303887918999997, 2024-08-30T16:04:53.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.10442118, 2024-08-30T16:04:53.206 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:53.206 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:53.206 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:53.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:53.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:53.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:53.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:53.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895819+0000", 2024-08-30T16:04:53.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:53.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:53.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.895800+0000", 2024-08-30T16:04:53.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999809 2024-08-30T16:04:53.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:53.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.921337+0000", 2024-08-30T16:04:53.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.02553677 2024-08-30T16:04:53.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:53.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.926424+0000", 2024-08-30T16:04:53.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050871340000000001 2024-08-30T16:04:53.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:53.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.935234+0000", 2024-08-30T16:04:53.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088098540000000006 2024-08-30T16:04:53.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:53.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.944321+0000", 2024-08-30T16:04:53.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0090873329999999995 2024-08-30T16:04:53.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:53.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.949613+0000", 2024-08-30T16:04:53.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052919710000000003 2024-08-30T16:04:53.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:53.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:08.000240+0000", 2024-08-30T16:04:53.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.050627411999999997 2024-08-30T16:04:53.212 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:53.212 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:53.212 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:53.212 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:53.212 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:53.212 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:53.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T16:04:53.291 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-08-30T16:04:53.292 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T16:04:53.292 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:53.290+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:53.342 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:53.342 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:53.342 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:53.342 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:53.352 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-08-30T16:04:53.382 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-08-30T16:04:53.382 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T16:04:53.391 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:53.389+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:53.476 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:16.081 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:53.479 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:16.085 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:53.501 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-08-30T16:04:53.501 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T16:04:53.502 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:53.500+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:53.503 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:16.108 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:53.526 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:15.063 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:53.529 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:15.066 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:53.529 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:53.529 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:53.529 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:53.530 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:53.530 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:53.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-08-30T16:04:53.547 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:15.084 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:53.550 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:14.709 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:53.552 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:14.711 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:53.566 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:14.726 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:04:53.588 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-08-30T16:04:53.588 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T16:04:53.613 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:53.612+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:53.687 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:53.687 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:53.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:53.687 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:53.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.687 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:53.688 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:53.688 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 332.41072786000001, 2024-08-30T16:04:53.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.207928909, 2024-08-30T16:04:53.688 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:53.688 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:53.688 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:53.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:53.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:53.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:53.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:53.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268967+0000", 2024-08-30T16:04:53.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:53.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:53.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.268914+0000", 2024-08-30T16:04:53.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999471 2024-08-30T16:04:53.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:53.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.295474+0000", 2024-08-30T16:04:53.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026559225999999998 2024-08-30T16:04:53.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:53.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.298120+0000", 2024-08-30T16:04:53.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026460070000000001 2024-08-30T16:04:53.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:53.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.306997+0000", 2024-08-30T16:04:53.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088774679999999995 2024-08-30T16:04:53.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:53.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.315737+0000", 2024-08-30T16:04:53.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087397559999999996 2024-08-30T16:04:53.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:53.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.321150+0000", 2024-08-30T16:04:53.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054129249999999999 2024-08-30T16:04:53.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:53.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.476896+0000", 2024-08-30T16:04:53.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.155746622 2024-08-30T16:04:53.693 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:53.693 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:53.693 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:53.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.693 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(1.0 23/22 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:53.693 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:53.693 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 332.13780049899998, 2024-08-30T16:04:53.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033462549000000001, 2024-08-30T16:04:53.694 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:53.694 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:53.694 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:53.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:53.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:53.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:53.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:53.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541895+0000", 2024-08-30T16:04:53.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:53.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:53.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.541872+0000", 2024-08-30T16:04:53.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999776 2024-08-30T16:04:53.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:53.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.546883+0000", 2024-08-30T16:04:53.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005010876 2024-08-30T16:04:53.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:53.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547174+0000", 2024-08-30T16:04:53.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029043899999999997 2024-08-30T16:04:53.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:53.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.547518+0000", 2024-08-30T16:04:53.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000344106 2024-08-30T16:04:53.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:53.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549055+0000", 2024-08-30T16:04:53.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015375059999999999 2024-08-30T16:04:53.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:53.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.549352+0000", 2024-08-30T16:04:53.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000296772 2024-08-30T16:04:53.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:53.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:21.575357+0000", 2024-08-30T16:04:53.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026005118000000001 2024-08-30T16:04:53.698 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:53.698 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:53.699 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:53.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.699 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPull(2.3 e83/82 cost 1049576)", 2024-08-30T16:04:53.699 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:53.699 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 45.853714261, 2024-08-30T16:04:53.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051106587000000002, 2024-08-30T16:04:53.699 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:53.699 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:53.699 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:53.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:53.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:53.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:53.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:53.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825981+0000", 2024-08-30T16:04:53.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:53.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:53.701 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.825962+0000", 2024-08-30T16:04:53.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:53.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:53.701 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.833441+0000", 2024-08-30T16:04:53.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0074797580000000004 2024-08-30T16:04:53.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:53.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.833651+0000", 2024-08-30T16:04:53.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000209414 2024-08-30T16:04:53.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:53.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.834053+0000", 2024-08-30T16:04:53.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040233199999999998 2024-08-30T16:04:53.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:53.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.835125+0000", 2024-08-30T16:04:53.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010716930000000001 2024-08-30T16:04:53.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:53.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:53.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:53.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:07.877088+0000", 2024-08-30T16:04:53.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.041962869999999999 2024-08-30T16:04:53.703 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:53.703 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:53.703 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:53.703 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:53.703 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:53.703 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:53.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-08-30T16:04:53.705 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-08-30T16:04:53.706 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T16:04:53.706 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:53.703+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:53.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-30T16:04:53.806 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:53.804+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:53.867 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-30T16:04:53.895 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:53.895 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:53.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-08-30T16:04:53.907 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:04:53.905+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:04:53.966 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-08-30T16:04:53.966 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T16:04:53.970 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T16:04:53.971 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T16:04:54.007 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:04:54.006+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:04:54.081 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:54.081 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:54.081 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:54.082 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:54.082 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:54.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-08-30T16:04:54.134 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:04:54.133+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:04:54.209 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:04:54.207+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:04:54.228 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:54.228 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:54.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:54.228 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:54.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.229 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4288.0:1 2.3 e25/24 2:c4c92e5a:::rbd_trash:head v 25'1, mlcod=0'0)", 2024-08-30T16:04:54.229 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:54.229 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 328.84957885199998, 2024-08-30T16:04:54.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.163864855, 2024-08-30T16:04:54.229 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:54.229 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:54.229 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:54.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:54.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:54.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:54.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374185+0000", 2024-08-30T16:04:54.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:54.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.374168+0000", 2024-08-30T16:04:54.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999828 2024-08-30T16:04:54.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:54.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.402662+0000", 2024-08-30T16:04:54.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028494572999999999 2024-08-30T16:04:54.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:54.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.403722+0000", 2024-08-30T16:04:54.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001059485 2024-08-30T16:04:54.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:54.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.412505+0000", 2024-08-30T16:04:54.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087830549999999997 2024-08-30T16:04:54.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:54.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.420658+0000", 2024-08-30T16:04:54.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081527969999999998 2024-08-30T16:04:54.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:54.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.430692+0000", 2024-08-30T16:04:54.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010033752999999999 2024-08-30T16:04:54.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:54.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.525477+0000", 2024-08-30T16:04:54.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.094785764999999994 2024-08-30T16:04:54.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:54.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.538050+0000", 2024-08-30T16:04:54.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012572677000000001 2024-08-30T16:04:54.233 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.233 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.233 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.233 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.234 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.234 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:54.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-08-30T16:04:54.321 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:54.320+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:54.406 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:54.406 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:54.406 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-08-30T16:04:54.406 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:54.411 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:04:54.409+0000 adad640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:04:54.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-08-30T16:04:54.452 INFO:teuthology.orchestra.run.smithi050.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","modified":"2024-08-30T15:57:29.788167Z","created":"2024-08-30T15:57:29.788167Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-30T16:04:54.466 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-08-30T16:04:54.466 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-08-30T16:04:54.476 INFO:tasks.thrashosds:joining thrashosds 2024-08-30T16:04:54.582 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:54.583 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-08-30T16:04:54.583 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-08-30T16:04:54.583 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-08-30T16:04:54.583 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:54.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-08-30T16:04:54.625 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:04:54.715 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-08-30T16:04:54.718 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T16:04:54.718 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-08-30T16:04:54.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-08-30T16:04:54.749 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:04:54.749 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-08-30T16:04:54.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-08-30T16:04:54.749 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-08-30T16:04:54.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.749 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:54.749 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:54.750 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 330.29150875099998, 2024-08-30T16:04:54.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1530891640000001, 2024-08-30T16:04:54.750 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:54.750 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:54.750 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:54.750 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:54.750 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:54.750 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:54.750 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.750 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:54.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.750 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:54.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:54.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:54.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426735+0000", 2024-08-30T16:04:54.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:54.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.426232+0000", 2024-08-30T16:04:54.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9994974 2024-08-30T16:04:54.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:54.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.431118+0000", 2024-08-30T16:04:54.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048865109999999996 2024-08-30T16:04:54.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:54.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.435506+0000", 2024-08-30T16:04:54.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043877020000000003 2024-08-30T16:04:54.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:54.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.444441+0000", 2024-08-30T16:04:54.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089344090000000008 2024-08-30T16:04:54.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:54.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.462730+0000", 2024-08-30T16:04:54.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018289041999999998 2024-08-30T16:04:54.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-08-30T16:04:54.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:24.469038+0000", 2024-08-30T16:04:54.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006308681 2024-08-30T16:04:54.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:54.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.033879+0000", 2024-08-30T16:04:54.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56484067100000002 2024-08-30T16:04:54.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:54.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.119521+0000", 2024-08-30T16:04:54.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085641816999999995 2024-08-30T16:04:54.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2", 2024-08-30T16:04:54.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.223673+0000", 2024-08-30T16:04:54.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.104152487 2024-08-30T16:04:54.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:54.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.360155+0000", 2024-08-30T16:04:54.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.136481563 2024-08-30T16:04:54.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:54.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.552403+0000", 2024-08-30T16:04:54.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19224792800000001 2024-08-30T16:04:54.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:54.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.562907+0000", 2024-08-30T16:04:54.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504629999999999 2024-08-30T16:04:54.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:54.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.579824+0000", 2024-08-30T16:04:54.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016916339999999998 2024-08-30T16:04:54.756 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.756 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.756 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.757 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:1 2.3 e25/24 ondisk, result = 0)", 2024-08-30T16:04:54.757 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:54.757 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 329.17564241600002, 2024-08-30T16:04:54.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042434870999999999, 2024-08-30T16:04:54.757 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:54.757 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:54.757 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:54.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:54.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:54.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:54.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542601+0000", 2024-08-30T16:04:54.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:54.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.542583+0000", 2024-08-30T16:04:54.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:54.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:54.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.544470+0000", 2024-08-30T16:04:54.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001887562 2024-08-30T16:04:54.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:54.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545613+0000", 2024-08-30T16:04:54.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011429439999999999 2024-08-30T16:04:54.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:54.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.545979+0000", 2024-08-30T16:04:54.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036620499999999998 2024-08-30T16:04:54.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:54.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.547085+0000", 2024-08-30T16:04:54.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011056530000000001 2024-08-30T16:04:54.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:54.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.550689+0000", 2024-08-30T16:04:54.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036035820000000001 2024-08-30T16:04:54.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:54.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.585036+0000", 2024-08-30T16:04:54.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034347063999999997 2024-08-30T16:04:54.760 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.760 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.761 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.761 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25)", 2024-08-30T16:04:54.761 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:54.761 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 329.11605091799998, 2024-08-30T16:04:54.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022242349000000002, 2024-08-30T16:04:54.761 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:54.761 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:54.761 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:54.761 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:54.761 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:54.761 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-08-30T16:04:54.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.762 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:54.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:54.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:54.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:54.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602192+0000", 2024-08-30T16:04:54.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:54.763 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.602173+0000", 2024-08-30T16:04:54.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999804 2024-08-30T16:04:54.763 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:54.763 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.605346+0000", 2024-08-30T16:04:54.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003173375 2024-08-30T16:04:54.763 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:54.763 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.606897+0000", 2024-08-30T16:04:54.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015506700000000001 2024-08-30T16:04:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.607171+0000", 2024-08-30T16:04:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000273983 2024-08-30T16:04:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.609153+0000", 2024-08-30T16:04:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019820290000000002 2024-08-30T16:04:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:54.765 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.624435+0000", 2024-08-30T16:04:54.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015281747 2024-08-30T16:04:54.765 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.765 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.765 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.765 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-08-30T16:04:54.765 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:54.765 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 329.08935223100002, 2024-08-30T16:04:54.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082245289999999995, 2024-08-30T16:04:54.765 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:54.765 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:54.766 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:54.766 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:54.766 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:54.766 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:54.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.766 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:54.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.766 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:54.766 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:54.766 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.766 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:54.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628891+0000", 2024-08-30T16:04:54.767 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:54.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628871+0000", 2024-08-30T16:04:54.767 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:54.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:54.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.628956+0000", 2024-08-30T16:04:54.767 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5616000000000002e-05 2024-08-30T16:04:54.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629134+0000", 2024-08-30T16:04:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017781399999999999 2024-08-30T16:04:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.629299+0000", 2024-08-30T16:04:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016502199999999999 2024-08-30T16:04:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.632827+0000", 2024-08-30T16:04:54.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003527723 2024-08-30T16:04:54.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.769 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:54.769 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:25.637116+0000", 2024-08-30T16:04:54.769 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042887350000000001 2024-08-30T16:04:54.769 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.769 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.769 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.769 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:54.769 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:54.770 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 328.22173570299998, 2024-08-30T16:04:54.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16365439700000001, 2024-08-30T16:04:54.770 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:54.770 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:54.770 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:54.770 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:54.770 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:54.770 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:54.770 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.770 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:54.770 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496508+0000", 2024-08-30T16:04:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496483+0000", 2024-08-30T16:04:54.771 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496636+0000", 2024-08-30T16:04:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015275299999999999 2024-08-30T16:04:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.496853+0000", 2024-08-30T16:04:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021731800000000001 2024-08-30T16:04:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497062+0000", 2024-08-30T16:04:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000208755 2024-08-30T16:04:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.497886+0000", 2024-08-30T16:04:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082446699999999995 2024-08-30T16:04:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:54.773 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.500509+0000", 2024-08-30T16:04:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026228839999999998 2024-08-30T16:04:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.516558+0000", 2024-08-30T16:04:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016048687999999998 2024-08-30T16:04:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.527712+0000", 2024-08-30T16:04:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01115406 2024-08-30T16:04:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.657021+0000", 2024-08-30T16:04:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12930898199999999 2024-08-30T16:04:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.658671+0000", 2024-08-30T16:04:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016502520000000001 2024-08-30T16:04:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:54.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660162+0000", 2024-08-30T16:04:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014909859999999999 2024-08-30T16:04:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:5 2.2 e27/24 ondisk, result = 0)", 2024-08-30T16:04:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 328.06248862400003, 2024-08-30T16:04:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049179239999999997, 2024-08-30T16:04:54.776 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655755+0000", 2024-08-30T16:04:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655732+0000", 2024-08-30T16:04:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999771 2024-08-30T16:04:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655823+0000", 2024-08-30T16:04:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1080000000000002e-05 2024-08-30T16:04:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.655978+0000", 2024-08-30T16:04:54.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015456199999999999 2024-08-30T16:04:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656097+0000", 2024-08-30T16:04:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118892 2024-08-30T16:04:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656753+0000", 2024-08-30T16:04:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065658600000000002 2024-08-30T16:04:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.656899+0000", 2024-08-30T16:04:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014557599999999999 2024-08-30T16:04:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.660673+0000", 2024-08-30T16:04:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037739169999999999 2024-08-30T16:04:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.780 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 327.78631455800002, 2024-08-30T16:04:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035220270000000001, 2024-08-30T16:04:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-08-30T16:04:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:54.781 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931929+0000", 2024-08-30T16:04:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:54.782 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.931911+0000", 2024-08-30T16:04:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999819 2024-08-30T16:04:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932016+0000", 2024-08-30T16:04:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105157 2024-08-30T16:04:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932174+0000", 2024-08-30T16:04:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015862600000000001 2024-08-30T16:04:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.932336+0000", 2024-08-30T16:04:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016111599999999999 2024-08-30T16:04:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.933327+0000", 2024-08-30T16:04:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099147199999999997 2024-08-30T16:04:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:26.935451+0000", 2024-08-30T16:04:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021237690000000002 2024-08-30T16:04:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4288.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 327.12802297899998, 2024-08-30T16:04:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.099852467, 2024-08-30T16:04:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:54.785 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4288", 2024-08-30T16:04:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.50:0/2806343493", 2024-08-30T16:04:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7 2024-08-30T16:04:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590220+0000", 2024-08-30T16:04:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590195+0000", 2024-08-30T16:04:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590333+0000", 2024-08-30T16:04:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137501 2024-08-30T16:04:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590511+0000", 2024-08-30T16:04:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000178011 2024-08-30T16:04:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.590679+0000", 2024-08-30T16:04:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000168722 2024-08-30T16:04:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.594725+0000", 2024-08-30T16:04:54.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040454280000000002 2024-08-30T16:04:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.608239+0000", 2024-08-30T16:04:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013514061000000001 2024-08-30T16:04:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6", 2024-08-30T16:04:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.614396+0000", 2024-08-30T16:04:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061575409999999999 2024-08-30T16:04:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-08-30T16:04:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.651896+0000", 2024-08-30T16:04:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037499645999999998 2024-08-30T16:04:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-08-30T16:04:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687859+0000", 2024-08-30T16:04:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035962809999999998 2024-08-30T16:04:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-08-30T16:04:54.790 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.689267+0000", 2024-08-30T16:04:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407763 2024-08-30T16:04:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690073+0000", 2024-08-30T16:04:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080623499999999998 2024-08-30T16:04:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.791 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4288.0:7 2.2 e28/24 ondisk, result = 0)", 2024-08-30T16:04:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 327.03210969499997, 2024-08-30T16:04:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040304640000000001, 2024-08-30T16:04:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-08-30T16:04:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686134+0000", 2024-08-30T16:04:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686108+0000", 2024-08-30T16:04:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686241+0000", 2024-08-30T16:04:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013288500000000001 2024-08-30T16:04:54.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686489+0000", 2024-08-30T16:04:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024837699999999998 2024-08-30T16:04:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.686685+0000", 2024-08-30T16:04:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019559699999999999 2024-08-30T16:04:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687568+0000", 2024-08-30T16:04:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088281500000000001 2024-08-30T16:04:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-08-30T16:04:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.687718+0000", 2024-08-30T16:04:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015017599999999999 2024-08-30T16:04:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T15:59:27.690164+0000", 2024-08-30T16:04:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002446035 2024-08-30T16:04:54.795 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-08-30T16:04:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 267.780749184, 2024-08-30T16:04:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070140929999999999, 2024-08-30T16:04:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:54.796 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2024-08-30T16:04:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937494+0000", 2024-08-30T16:04:54.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937470+0000", 2024-08-30T16:04:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999762 2024-08-30T16:04:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937612+0000", 2024-08-30T16:04:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014164700000000001 2024-08-30T16:04:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.937842+0000", 2024-08-30T16:04:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022947500000000001 2024-08-30T16:04:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.938024+0000", 2024-08-30T16:04:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018226700000000001 2024-08-30T16:04:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.942243+0000", 2024-08-30T16:04:54.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042188130000000001 2024-08-30T16:04:54.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:54.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:00:26.944508+0000", 2024-08-30T16:04:54.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022656109999999998 2024-08-30T16:04:54.800 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.800 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.800 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.800 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-08-30T16:04:54.800 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:54.801 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 207.77164449700001, 2024-08-30T16:04:54.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058362079999999998, 2024-08-30T16:04:54.801 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:54.801 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:54.801 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:54.801 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:54.801 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:54.801 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-08-30T16:04:54.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.801 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:54.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.801 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:54.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:54.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:54.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946599+0000", 2024-08-30T16:04:54.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:54.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946557+0000", 2024-08-30T16:04:54.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.999958 2024-08-30T16:04:54.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.803 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:54.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.946814+0000", 2024-08-30T16:04:54.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025691800000000002 2024-08-30T16:04:54.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.803 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:54.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947094+0000", 2024-08-30T16:04:54.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027992099999999998 2024-08-30T16:04:54.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.803 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:54.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.947294+0000", 2024-08-30T16:04:54.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000200497 2024-08-30T16:04:54.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:54.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.950504+0000", 2024-08-30T16:04:54.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003209702 2024-08-30T16:04:54.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:54.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:01:26.952435+0000", 2024-08-30T16:04:54.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019309609999999999 2024-08-30T16:04:54.804 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.804 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.805 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.805 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4171.0:11 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-08-30T16:04:54.805 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-08-30T16:04:27.021982+0000", 2024-08-30T16:04:54.805 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 27.696261222, 2024-08-30T16:04:54.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0095695959999999997, 2024-08-30T16:04:54.805 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-08-30T16:04:54.805 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-08-30T16:04:54.805 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-08-30T16:04:54.805 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4171", 2024-08-30T16:04:54.805 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:54.806 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 11 2024-08-30T16:04:54.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.806 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-08-30T16:04:54.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-08-30T16:04:54.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.021982+0000", 2024-08-30T16:04:54.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-08-30T16:04:54.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.021982+0000", 2024-08-30T16:04:54.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-08-30T16:04:54.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.807 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-08-30T16:04:54.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.021955+0000", 2024-08-30T16:04:54.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4294967295.9999728 2024-08-30T16:04:54.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.807 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-08-30T16:04:54.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.022110+0000", 2024-08-30T16:04:54.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015461899999999999 2024-08-30T16:04:54.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.807 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-08-30T16:04:54.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.022347+0000", 2024-08-30T16:04:54.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023690199999999999 2024-08-30T16:04:54.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-08-30T16:04:54.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.022532+0000", 2024-08-30T16:04:54.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000185339 2024-08-30T16:04:54.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-08-30T16:04:54.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.028853+0000", 2024-08-30T16:04:54.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063211270000000002 2024-08-30T16:04:54.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-08-30T16:04:54.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-08-30T16:04:54.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-08-30T16:04:54.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-08-30T16:04:27.031552+0000", 2024-08-30T16:04:54.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026985770000000002 2024-08-30T16:04:54.809 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.809 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.809 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.809 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-08-30T16:04:54.809 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-08-30T16:04:54.809 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:04:54.810 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-08-30T16:04:54.949 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:54.949 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:54.949 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:54.949 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:54.958 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-08-30T16:04:55.096 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:55.097 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:55.097 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:55.097 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:55.097 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:55.106 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-08-30T16:04:55.230 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:55.231 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:55.231 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:55.231 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [] 2024-08-30T16:04:55.231 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:55.240 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-08-30T16:04:55.369 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:55.369 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:55.370 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:55.370 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:55.379 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-08-30T16:04:55.506 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:55.506 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:55.506 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:55.516 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-08-30T16:04:55.692 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:55.693 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:55.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:55.693 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:55.693 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.693 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:55.693 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:55.693 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 344.82971542000001, 2024-08-30T16:04:55.693 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.473997905, 2024-08-30T16:04:55.693 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.693 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.693 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.694 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.694 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.694 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 1 2024-08-30T16:04:55.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.694 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:55.694 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.694 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.694 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.694 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.694 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.812454+0000", 2024-08-30T16:04:55.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.811972+0000", 2024-08-30T16:04:55.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9995179 2024-08-30T16:04:55.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.695 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.695 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.819755+0000", 2024-08-30T16:04:55.695 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077833950000000002 2024-08-30T16:04:55.695 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.695 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.824325+0000", 2024-08-30T16:04:55.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045700910000000001 2024-08-30T16:04:55.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.833153+0000", 2024-08-30T16:04:55.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088271180000000001 2024-08-30T16:04:55.696 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.696 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.696 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.696 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.843893+0000", 2024-08-30T16:04:55.696 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010740875 2024-08-30T16:04:55.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "waiting for peered", 2024-08-30T16:04:55.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.854080+0000", 2024-08-30T16:04:55.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.010186565999999999 2024-08-30T16:04:55.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.697 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.697 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:10.946066+0000", 2024-08-30T16:04:55.697 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.091986156999999999 2024-08-30T16:04:55.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.034080+0000", 2024-08-30T16:04:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088013330000000001 2024-08-30T16:04:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.233273+0000", 2024-08-30T16:04:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19919362099999999 2024-08-30T16:04:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.250661+0000", 2024-08-30T16:04:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.017388329000000001 2024-08-30T16:04:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.699 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.699 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.286452+0000", 2024-08-30T16:04:55.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.035790534999999998 2024-08-30T16:04:55.699 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.699 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.699 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.699 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:20 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:55.699 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:55.699 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 343.91329940000003, 2024-08-30T16:04:55.699 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.19520754700000001, 2024-08-30T16:04:55.700 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.700 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.700 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.700 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.700 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.700 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 20 2024-08-30T16:04:55.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.700 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.700 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.700 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:55.700 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728870+0000", 2024-08-30T16:04:55.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.728848+0000", 2024-08-30T16:04:55.701 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:55.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.701 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.701 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730055+0000", 2024-08-30T16:04:55.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012065159999999999 2024-08-30T16:04:55.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730248+0000", 2024-08-30T16:04:55.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00019328699999999999 2024-08-30T16:04:55.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.702 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730408+0000", 2024-08-30T16:04:55.702 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016058500000000001 2024-08-30T16:04:55.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.702 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.731609+0000", 2024-08-30T16:04:55.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012007809999999999 2024-08-30T16:04:55.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.734561+0000", 2024-08-30T16:04:55.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029517609999999998 2024-08-30T16:04:55.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.703 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.703 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.919184+0000", 2024-08-30T16:04:55.703 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.18462294900000001 2024-08-30T16:04:55.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.921218+0000", 2024-08-30T16:04:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0020337480000000002 2024-08-30T16:04:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.924078+0000", 2024-08-30T16:04:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028599910000000001 2024-08-30T16:04:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.705 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-08-30T16:04:55.705 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:55.705 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 343.911594847, 2024-08-30T16:04:55.705 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.20569897100000001, 2024-08-30T16:04:55.705 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.705 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.705 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.705 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.705 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.705 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 21 2024-08-30T16:04:55.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.706 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:55.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730575+0000", 2024-08-30T16:04:55.706 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.706 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.706 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.730559+0000", 2024-08-30T16:04:55.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999843 2024-08-30T16:04:55.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860710+0000", 2024-08-30T16:04:55.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.130151132 2024-08-30T16:04:55.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.707 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.707 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.860882+0000", 2024-08-30T16:04:55.707 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017205399999999999 2024-08-30T16:04:55.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.861039+0000", 2024-08-30T16:04:55.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000156463 2024-08-30T16:04:55.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.708 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.708 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.862588+0000", 2024-08-30T16:04:55.708 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015493169999999999 2024-08-30T16:04:55.708 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.708 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.709 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.864437+0000", 2024-08-30T16:04:55.709 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001848616 2024-08-30T16:04:55.709 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.709 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.709 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.932049+0000", 2024-08-30T16:04:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.067611822000000002 2024-08-30T16:04:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.933568+0000", 2024-08-30T16:04:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015197330000000001 2024-08-30T16:04:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:11.936274+0000", 2024-08-30T16:04:55.710 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027053480000000002 2024-08-30T16:04:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 343.49214103999998, 2024-08-30T16:04:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.15598171199999999, 2024-08-30T16:04:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.711 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 30 2024-08-30T16:04:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150028+0000", 2024-08-30T16:04:55.712 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.149997+0000", 2024-08-30T16:04:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999685 2024-08-30T16:04:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150184+0000", 2024-08-30T16:04:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000186956 2024-08-30T16:04:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.713 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150443+0000", 2024-08-30T16:04:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025857000000000001 2024-08-30T16:04:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.150696+0000", 2024-08-30T16:04:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025305400000000001 2024-08-30T16:04:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.151378+0000", 2024-08-30T16:04:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00068193099999999999 2024-08-30T16:04:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.714 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.153038+0000", 2024-08-30T16:04:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001660075 2024-08-30T16:04:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.301689+0000", 2024-08-30T16:04:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148651792 2024-08-30T16:04:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.715 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.303256+0000", 2024-08-30T16:04:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015661500000000001 2024-08-30T16:04:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.306010+0000", 2024-08-30T16:04:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002754645 2024-08-30T16:04:55.716 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 343.47525286600001, 2024-08-30T16:04:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.147660558, 2024-08-30T16:04:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.717 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 31 2024-08-30T16:04:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.718 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166917+0000", 2024-08-30T16:04:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.166888+0000", 2024-08-30T16:04:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999709 2024-08-30T16:04:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167065+0000", 2024-08-30T16:04:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000177725 2024-08-30T16:04:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167373+0000", 2024-08-30T16:04:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00030805699999999998 2024-08-30T16:04:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.167634+0000", 2024-08-30T16:04:55.720 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026103200000000002 2024-08-30T16:04:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.168669+0000", 2024-08-30T16:04:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001034727 2024-08-30T16:04:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.170419+0000", 2024-08-30T16:04:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001749565 2024-08-30T16:04:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.721 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.310406+0000", 2024-08-30T16:04:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13998732999999999 2024-08-30T16:04:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.311877+0000", 2024-08-30T16:04:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001470956 2024-08-30T16:04:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314577+0000", 2024-08-30T16:04:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027002739999999999 2024-08-30T16:04:55.722 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 343.46225279700002, 2024-08-30T16:04:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13899110100000001, 2024-08-30T16:04:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.723 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 32 2024-08-30T16:04:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.724 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179917+0000", 2024-08-30T16:04:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.179888+0000", 2024-08-30T16:04:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999719 2024-08-30T16:04:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181661+0000", 2024-08-30T16:04:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017728760000000001 2024-08-30T16:04:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.725 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.181927+0000", 2024-08-30T16:04:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000265851 2024-08-30T16:04:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182186+0000", 2024-08-30T16:04:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025863699999999998 2024-08-30T16:04:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.189408+0000", 2024-08-30T16:04:55.726 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0072222830000000003 2024-08-30T16:04:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.192358+0000", 2024-08-30T16:04:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029503760000000002 2024-08-30T16:04:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.314819+0000", 2024-08-30T16:04:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122460916 2024-08-30T16:04:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.727 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.316250+0000", 2024-08-30T16:04:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430284 2024-08-30T16:04:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.318908+0000", 2024-08-30T16:04:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026581909999999999 2024-08-30T16:04:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.728 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 343.45976050500002, 2024-08-30T16:04:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.140851383, 2024-08-30T16:04:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 33 2024-08-30T16:04:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.729 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182409+0000", 2024-08-30T16:04:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182384+0000", 2024-08-30T16:04:55.730 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999752 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182523+0000", 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000138937 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182743+0000", 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022047000000000001 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.182977+0000", 2024-08-30T16:04:55.731 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023332100000000001 2024-08-30T16:04:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.212206+0000", 2024-08-30T16:04:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029229627000000001 2024-08-30T16:04:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.215261+0000", 2024-08-30T16:04:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0030546520000000001 2024-08-30T16:04:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.732 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.319151+0000", 2024-08-30T16:04:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.103890204 2024-08-30T16:04:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.320581+0000", 2024-08-30T16:04:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001430185 2024-08-30T16:04:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323260+0000", 2024-08-30T16:04:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026789579999999999 2024-08-30T16:04:55.733 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 343.45900305700002, 2024-08-30T16:04:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14446492499999999, 2024-08-30T16:04:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.734 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 34 2024-08-30T16:04:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183166+0000", 2024-08-30T16:04:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.735 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183147+0000", 2024-08-30T16:04:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.99998 2024-08-30T16:04:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183271+0000", 2024-08-30T16:04:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124912 2024-08-30T16:04:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183485+0000", 2024-08-30T16:04:55.736 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000213193 2024-08-30T16:04:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183715+0000", 2024-08-30T16:04:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023065499999999999 2024-08-30T16:04:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.231073+0000", 2024-08-30T16:04:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.047357248999999997 2024-08-30T16:04:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.737 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.233989+0000", 2024-08-30T16:04:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029165179999999999 2024-08-30T16:04:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.323513+0000", 2024-08-30T16:04:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.089523569999999997 2024-08-30T16:04:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.324934+0000", 2024-08-30T16:04:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00142121 2024-08-30T16:04:55.738 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327631+0000", 2024-08-30T16:04:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026975609999999998 2024-08-30T16:04:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 343.45827060699997, 2024-08-30T16:04:55.739 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14805279700000001, 2024-08-30T16:04:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 35 2024-08-30T16:04:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.740 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183899+0000", 2024-08-30T16:04:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.183877+0000", 2024-08-30T16:04:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999781 2024-08-30T16:04:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.741 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.185770+0000", 2024-08-30T16:04:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001892868 2024-08-30T16:04:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186010+0000", 2024-08-30T16:04:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023985400000000001 2024-08-30T16:04:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186253+0000", 2024-08-30T16:04:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024310999999999999 2024-08-30T16:04:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.742 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.248688+0000", 2024-08-30T16:04:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.062435280000000003 2024-08-30T16:04:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.251574+0000", 2024-08-30T16:04:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028856630000000001 2024-08-30T16:04:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.743 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.327860+0000", 2024-08-30T16:04:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.076286463999999998 2024-08-30T16:04:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.329281+0000", 2024-08-30T16:04:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001420665 2024-08-30T16:04:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.331952+0000", 2024-08-30T16:04:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026706630000000002 2024-08-30T16:04:55.744 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 343.455731808, 2024-08-30T16:04:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.14981608599999999, 2024-08-30T16:04:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.745 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 36 2024-08-30T16:04:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186438+0000", 2024-08-30T16:04:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.746 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186417+0000", 2024-08-30T16:04:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186547+0000", 2024-08-30T16:04:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013036699999999999 2024-08-30T16:04:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186768+0000", 2024-08-30T16:04:55.747 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022107999999999999 2024-08-30T16:04:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.186999+0000", 2024-08-30T16:04:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000231294 2024-08-30T16:04:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.262316+0000", 2024-08-30T16:04:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.075317099999999998 2024-08-30T16:04:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.748 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.264028+0000", 2024-08-30T16:04:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001711887 2024-08-30T16:04:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.332189+0000", 2024-08-30T16:04:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.068160525999999999 2024-08-30T16:04:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.333601+0000", 2024-08-30T16:04:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001412052 2024-08-30T16:04:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.749 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336254+0000", 2024-08-30T16:04:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0026530009999999999 2024-08-30T16:04:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 343.45498380700002, 2024-08-30T16:04:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.153348177, 2024-08-30T16:04:55.750 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 37 2024-08-30T16:04:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.751 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187186+0000", 2024-08-30T16:04:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187165+0000", 2024-08-30T16:04:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.999979 2024-08-30T16:04:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187295+0000", 2024-08-30T16:04:55.752 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013017399999999999 2024-08-30T16:04:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187514+0000", 2024-08-30T16:04:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00021951100000000001 2024-08-30T16:04:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.187739+0000", 2024-08-30T16:04:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00022512699999999999 2024-08-30T16:04:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.753 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.270864+0000", 2024-08-30T16:04:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.083124898000000003 2024-08-30T16:04:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.272638+0000", 2024-08-30T16:04:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017731660000000001 2024-08-30T16:04:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.336492+0000", 2024-08-30T16:04:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.063854772000000004 2024-08-30T16:04:55.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.337958+0000", 2024-08-30T16:04:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00146535 2024-08-30T16:04:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.340534+0000", 2024-08-30T16:04:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002576259 2024-08-30T16:04:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 342.68286733500003, 2024-08-30T16:04:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115515616, 2024-08-30T16:04:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 72 2024-08-30T16:04:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.756 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959302+0000", 2024-08-30T16:04:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.757 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.959279+0000", 2024-08-30T16:04:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960667+0000", 2024-08-30T16:04:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013878930000000001 2024-08-30T16:04:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.960913+0000", 2024-08-30T16:04:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024668500000000001 2024-08-30T16:04:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.758 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961164+0000", 2024-08-30T16:04:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025088900000000002 2024-08-30T16:04:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.987083+0000", 2024-08-30T16:04:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.025918566000000001 2024-08-30T16:04:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.989937+0000", 2024-08-30T16:04:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028542260000000001 2024-08-30T16:04:55.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.070468+0000", 2024-08-30T16:04:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080530688000000003 2024-08-30T16:04:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.071998+0000", 2024-08-30T16:04:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001529915 2024-08-30T16:04:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.760 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.074818+0000", 2024-08-30T16:04:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028202650000000002 2024-08-30T16:04:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 342.680810189, 2024-08-30T16:04:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.1183164, 2024-08-30T16:04:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.761 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 73 2024-08-30T16:04:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961359+0000", 2024-08-30T16:04:55.762 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:12.961334+0000", 2024-08-30T16:04:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999747 2024-08-30T16:04:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004622+0000", 2024-08-30T16:04:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.043288168000000002 2024-08-30T16:04:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.763 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.004880+0000", 2024-08-30T16:04:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025736200000000002 2024-08-30T16:04:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.005276+0000", 2024-08-30T16:04:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000396411 2024-08-30T16:04:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.013517+0000", 2024-08-30T16:04:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0082413669999999994 2024-08-30T16:04:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.764 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.016414+0000", 2024-08-30T16:04:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028964080000000001 2024-08-30T16:04:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.075389+0000", 2024-08-30T16:04:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.058975455000000003 2024-08-30T16:04:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.076875+0000", 2024-08-30T16:04:55.765 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0014855739999999999 2024-08-30T16:04:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.079676+0000", 2024-08-30T16:04:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028010040000000002 2024-08-30T16:04:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 342.417865689, 2024-08-30T16:04:55.766 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.148634612, 2024-08-30T16:04:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 82 2024-08-30T16:04:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224304+0000", 2024-08-30T16:04:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224275+0000", 2024-08-30T16:04:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999714 2024-08-30T16:04:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224448+0000", 2024-08-30T16:04:55.768 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017279099999999999 2024-08-30T16:04:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224698+0000", 2024-08-30T16:04:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025022299999999999 2024-08-30T16:04:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.224945+0000", 2024-08-30T16:04:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00024656 2024-08-30T16:04:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.769 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.227275+0000", 2024-08-30T16:04:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0023296599999999999 2024-08-30T16:04:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.230151+0000", 2024-08-30T16:04:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028762890000000002 2024-08-30T16:04:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.367647+0000", 2024-08-30T16:04:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13749620200000001 2024-08-30T16:04:55.770 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.370165+0000", 2024-08-30T16:04:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0025176869999999998 2024-08-30T16:04:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.372938+0000", 2024-08-30T16:04:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0027736380000000001 2024-08-30T16:04:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.771 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 342.416774186, 2024-08-30T16:04:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.151922004, 2024-08-30T16:04:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.772 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 83 2024-08-30T16:04:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225395+0000", 2024-08-30T16:04:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.773 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225374+0000", 2024-08-30T16:04:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999785 2024-08-30T16:04:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225499+0000", 2024-08-30T16:04:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124549 2024-08-30T16:04:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225733+0000", 2024-08-30T16:04:55.774 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023475699999999999 2024-08-30T16:04:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.225960+0000", 2024-08-30T16:04:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000226839 2024-08-30T16:04:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.240850+0000", 2024-08-30T16:04:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014890304 2024-08-30T16:04:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.775 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.243764+0000", 2024-08-30T16:04:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029136890000000001 2024-08-30T16:04:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.373189+0000", 2024-08-30T16:04:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.12942527600000001 2024-08-30T16:04:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.374612+0000", 2024-08-30T16:04:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001422796 2024-08-30T16:04:55.776 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.377317+0000", 2024-08-30T16:04:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002705182 2024-08-30T16:04:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.777 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 342.416024601, 2024-08-30T16:04:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.163237036, 2024-08-30T16:04:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 84 2024-08-30T16:04:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226145+0000", 2024-08-30T16:04:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.226124+0000", 2024-08-30T16:04:55.779 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257768+0000", 2024-08-30T16:04:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.031643522 2024-08-30T16:04:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.257950+0000", 2024-08-30T16:04:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00018222 2024-08-30T16:04:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.780 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.258123+0000", 2024-08-30T16:04:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00017252500000000001 2024-08-30T16:04:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.261994+0000", 2024-08-30T16:04:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0038712659999999999 2024-08-30T16:04:55.781 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.264861+0000", 2024-08-30T16:04:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028666149999999999 2024-08-30T16:04:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.386875+0000", 2024-08-30T16:04:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.122014337 2024-08-30T16:04:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.782 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.387756+0000", 2024-08-30T16:04:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00088113099999999995 2024-08-30T16:04:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389382+0000", 2024-08-30T16:04:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001625991 2024-08-30T16:04:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:55.783 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 342.38229306900001, 2024-08-30T16:04:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13203236099999999, 2024-08-30T16:04:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 85 2024-08-30T16:04:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:55.784 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259876+0000", 2024-08-30T16:04:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.785 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259850+0000", 2024-08-30T16:04:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999738 2024-08-30T16:04:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.259966+0000", 2024-08-30T16:04:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000116217 2024-08-30T16:04:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260135+0000", 2024-08-30T16:04:55.786 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00016898400000000001 2024-08-30T16:04:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260286+0000", 2024-08-30T16:04:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015051299999999999 2024-08-30T16:04:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.276722+0000", 2024-08-30T16:04:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.016436243999999999 2024-08-30T16:04:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.787 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.279555+0000", 2024-08-30T16:04:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028326219999999999 2024-08-30T16:04:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.389530+0000", 2024-08-30T16:04:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.10997583800000001 2024-08-30T16:04:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.390350+0000", 2024-08-30T16:04:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081993099999999998 2024-08-30T16:04:55.788 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.391909+0000", 2024-08-30T16:04:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0015584049999999999 2024-08-30T16:04:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 342.38176706100001, 2024-08-30T16:04:55.789 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.133994696, 2024-08-30T16:04:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 86 2024-08-30T16:04:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.790 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260402+0000", 2024-08-30T16:04:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260389+0000", 2024-08-30T16:04:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.791 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260473+0000", 2024-08-30T16:04:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 8.4478e-05 2024-08-30T16:04:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260608+0000", 2024-08-30T16:04:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013491299999999999 2024-08-30T16:04:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260748+0000", 2024-08-30T16:04:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013947300000000001 2024-08-30T16:04:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.792 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.290211+0000", 2024-08-30T16:04:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.029463360000000001 2024-08-30T16:04:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.293036+0000", 2024-08-30T16:04:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.002825036 2024-08-30T16:04:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392051+0000", 2024-08-30T16:04:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.099015293000000004 2024-08-30T16:04:55.793 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.392868+0000", 2024-08-30T16:04:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081691000000000001 2024-08-30T16:04:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394397+0000", 2024-08-30T16:04:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001528988 2024-08-30T16:04:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.794 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-08-30T16:04:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 342.381311498, 2024-08-30T16:04:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.13601835200000001, 2024-08-30T16:04:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 87 2024-08-30T16:04:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.795 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260858+0000", 2024-08-30T16:04:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.260845+0000", 2024-08-30T16:04:55.796 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999866 2024-08-30T16:04:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.303818+0000", 2024-08-30T16:04:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.042973624000000002 2024-08-30T16:04:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304079+0000", 2024-08-30T16:04:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00026066700000000002 2024-08-30T16:04:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.797 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.304335+0000", 2024-08-30T16:04:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00025559699999999999 2024-08-30T16:04:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.311317+0000", 2024-08-30T16:04:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0069824739999999998 2024-08-30T16:04:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.314238+0000", 2024-08-30T16:04:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0029208250000000002 2024-08-30T16:04:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.798 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.394534+0000", 2024-08-30T16:04:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.080296118999999999 2024-08-30T16:04:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.395354+0000", 2024-08-30T16:04:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00081939000000000003 2024-08-30T16:04:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:13.396876+0000", 2024-08-30T16:04:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001522863 2024-08-30T16:04:55.799 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4165.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-08-30T16:04:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 337.71079618200002, 2024-08-30T16:04:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.115784765, 2024-08-30T16:04:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T16:04:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4165", 2024-08-30T16:04:55.800 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/4205708399", 2024-08-30T16:04:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 106 2024-08-30T16:04:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931373+0000", 2024-08-30T16:04:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:55.801 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.931359+0000", 2024-08-30T16:04:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999862 2024-08-30T16:04:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.951988+0000", 2024-08-30T16:04:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.020628474000000001 2024-08-30T16:04:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952161+0000", 2024-08-30T16:04:55.802 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000173005 2024-08-30T16:04:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.952318+0000", 2024-08-30T16:04:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00015702899999999999 2024-08-30T16:04:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.954236+0000", 2024-08-30T16:04:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001917871 2024-08-30T16:04:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.803 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:17.955949+0000", 2024-08-30T16:04:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017129039999999999 2024-08-30T16:04:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "op_commit", 2024-08-30T16:04:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.044772+0000", 2024-08-30T16:04:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.088823692999999995 2024-08-30T16:04:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "commit_sent", 2024-08-30T16:04:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.045618+0000", 2024-08-30T16:04:55.804 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00084519200000000004 2024-08-30T16:04:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:18.047158+0000", 2024-08-30T16:04:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001540549 2024-08-30T16:04:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:55.805 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:55.805 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:55.806 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-08-30T16:04:55.931 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:55.931 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:55.932 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:04:55.932 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:55.940 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-08-30T16:04:56.042 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-08-30T16:04:56.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-08-30T16:04:56.079 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:56.079 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:04:56.079 INFO:teuthology.orchestra.run.smithi094.stdout: "complaint_time": 30, 2024-08-30T16:04:56.080 INFO:teuthology.orchestra.run.smithi094.stdout: "num_blocked_ops": 0 2024-08-30T16:04:56.080 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:56.089 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-08-30T16:04:56.260 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:04:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: "size": 20, 2024-08-30T16:04:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 600, 2024-08-30T16:04:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [ 2024-08-30T16:04:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-08-30T16:04:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:56.260 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 327.866307382, 2024-08-30T16:04:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.092299291000000006, 2024-08-30T16:04:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 2 2024-08-30T16:04:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:56.261 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.375156+0000", 2024-08-30T16:04:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.374653+0000", 2024-08-30T16:04:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9994969 2024-08-30T16:04:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.262 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.382397+0000", 2024-08-30T16:04:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0077442520000000001 2024-08-30T16:04:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.386691+0000", 2024-08-30T16:04:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.004293753 2024-08-30T16:04:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.396207+0000", 2024-08-30T16:04:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0095160169999999999 2024-08-30T16:04:56.263 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.410806+0000", 2024-08-30T16:04:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.014599186 2024-08-30T16:04:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T15:59:28.467456+0000", 2024-08-30T16:04:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056649315999999998 2024-08-30T16:04:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:56.264 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-08-30T16:04:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 207.753311266, 2024-08-30T16:04:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0028133139999999999, 2024-08-30T16:04:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:56.265 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 6 2024-08-30T16:04:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488152+0000", 2024-08-30T16:04:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:56.266 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488132+0000", 2024-08-30T16:04:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999795 2024-08-30T16:04:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488262+0000", 2024-08-30T16:04:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013008900000000001 2024-08-30T16:04:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:56.267 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488458+0000", 2024-08-30T16:04:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000195995 2024-08-30T16:04:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.488622+0000", 2024-08-30T16:04:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000163571 2024-08-30T16:04:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.489482+0000", 2024-08-30T16:04:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00086010199999999996 2024-08-30T16:04:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.268 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:01:28.490966+0000", 2024-08-30T16:04:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001484085 2024-08-30T16:04:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "osd_op(client.4171.0:10 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-08-30T16:04:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 87.744891894999995, 2024-08-30T16:04:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0037103309999999999, 2024-08-30T16:04:56.269 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "client_info": { 2024-08-30T16:04:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "client": "client.4171", 2024-08-30T16:04:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "client_addr": "v1:172.21.15.94:0/108091667", 2024-08-30T16:04:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "tid": 10 2024-08-30T16:04:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:56.270 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496572+0000", 2024-08-30T16:04:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496549+0000", 2024-08-30T16:04:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999766 2024-08-30T16:04:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496682+0000", 2024-08-30T16:04:56.271 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00013371699999999999 2024-08-30T16:04:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.496889+0000", 2024-08-30T16:04:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00020673600000000001 2024-08-30T16:04:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.497052+0000", 2024-08-30T16:04:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000162578 2024-08-30T16:04:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.272 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.498833+0000", 2024-08-30T16:04:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0017810530000000001 2024-08-30T16:04:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:03:28.500282+0000", 2024-08-30T16:04:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001449437 2024-08-30T16:04:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.273 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(2.3 83/82 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 25'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@25'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.363847864, 2024-08-30T16:04:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0088711689999999999, 2024-08-30T16:04:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.274 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877616+0000", 2024-08-30T16:04:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877588+0000", 2024-08-30T16:04:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999723 2024-08-30T16:04:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.275 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.877978+0000", 2024-08-30T16:04:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00038947500000000002 2024-08-30T16:04:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.878456+0000", 2024-08-30T16:04:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00047795699999999999 2024-08-30T16:04:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.878946+0000", 2024-08-30T16:04:56.276 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000490235 2024-08-30T16:04:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.880512+0000", 2024-08-30T16:04:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001565484 2024-08-30T16:04:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.880749+0000", 2024-08-30T16:04:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000237712 2024-08-30T16:04:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.277 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:07.886487+0000", 2024-08-30T16:04:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0057377349999999999 2024-08-30T16:04:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPushReply(2.3 83/82 [PushReplyOp(2:c4c92e5a:::rbd_trash:head)])", 2024-08-30T16:04:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 48.216440554000002, 2024-08-30T16:04:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0062707819999999999, 2024-08-30T16:04:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:56.278 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "reached pg", 2024-08-30T16:04:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:56.279 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025023+0000", 2024-08-30T16:04:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025012+0000", 2024-08-30T16:04:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999895 2024-08-30T16:04:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025103+0000", 2024-08-30T16:04:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 9.0877000000000004e-05 2024-08-30T16:04:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.280 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025231+0000", 2024-08-30T16:04:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00012737499999999999 2024-08-30T16:04:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.025355+0000", 2024-08-30T16:04:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000124194 2024-08-30T16:04:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.026750+0000", 2024-08-30T16:04:56.281 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001394922 2024-08-30T16:04:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:08.031294+0000", 2024-08-30T16:04:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0045441270000000002 2024-08-30T16:04:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.0 91/90 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 21'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@21'75, size: 131600, copy_subset: [0~131600], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:26.829025+0000", 2024-08-30T16:04:56.282 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.412438466000001, 2024-08-30T16:04:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.060649472000000003, 2024-08-30T16:04:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.829025+0000", 2024-08-30T16:04:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.283 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.829025+0000", 2024-08-30T16:04:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.828966+0000", 2024-08-30T16:04:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999404 2024-08-30T16:04:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:56.284 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.830358+0000", 2024-08-30T16:04:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0013920580000000001 2024-08-30T16:04:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.830913+0000", 2024-08-30T16:04:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00055505300000000001 2024-08-30T16:04:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.831281+0000", 2024-08-30T16:04:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000368193 2024-08-30T16:04:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.285 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.832628+0000", 2024-08-30T16:04:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.001346863 2024-08-30T16:04:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.833344+0000", 2024-08-30T16:04:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00071610100000000002 2024-08-30T16:04:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:26.889675+0000", 2024-08-30T16:04:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.056330587000000001 2024-08-30T16:04:56.286 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: "description": "MOSDPGPush(1.0 91/90 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 21'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@21'76, size: 327680, copy_subset: [0~327680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:327680, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-30T16:04:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: "initiated_at": "2024-08-30T16:04:27.013394+0000", 2024-08-30T16:04:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: "age": 29.228069987000001, 2024-08-30T16:04:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.021968463000000001, 2024-08-30T16:04:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: "type_data": { 2024-08-30T16:04:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: "flag_point": "started", 2024-08-30T16:04:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: "events": [ 2024-08-30T16:04:56.287 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "initiated", 2024-08-30T16:04:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.013394+0000", 2024-08-30T16:04:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "throttled", 2024-08-30T16:04:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.013394+0000", 2024-08-30T16:04:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0 2024-08-30T16:04:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "header_read", 2024-08-30T16:04:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.013362+0000", 2024-08-30T16:04:56.288 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 4294967295.9999681 2024-08-30T16:04:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "all_read", 2024-08-30T16:04:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.017776+0000", 2024-08-30T16:04:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0044143150000000003 2024-08-30T16:04:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "dispatched", 2024-08-30T16:04:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.018020+0000", 2024-08-30T16:04:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.000244076 2024-08-30T16:04:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.289 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "queued_for_pg", 2024-08-30T16:04:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.018340+0000", 2024-08-30T16:04:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00032027100000000001 2024-08-30T16:04:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "reached_pg", 2024-08-30T16:04:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.019618+0000", 2024-08-30T16:04:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.0012770640000000001 2024-08-30T16:04:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "started", 2024-08-30T16:04:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.019851+0000", 2024-08-30T16:04:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.00023378 2024-08-30T16:04:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-08-30T16:04:56.290 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-08-30T16:04:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: "event": "done", 2024-08-30T16:04:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: "time": "2024-08-30T16:04:27.035362+0000", 2024-08-30T16:04:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: "duration": 0.015510896999999999 2024-08-30T16:04:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-08-30T16:04:56.291 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-08-30T16:04:56.291 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:04:56.480 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:06.889 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:04:56.538 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:06.947 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:04:57.094 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-08-30T16:04:57.111 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:04:57.104+0000 5ef1600 -1 Falling back to public interface 2024-08-30T16:04:57.538 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:07.947 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:04:57.918 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:08.326 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:04:58.250 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:08.658 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:04:58.537 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:08.945 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:04:58.823 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:09.232 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:04:59.123 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:09.531 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:04:59.413 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:09.822 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:04:59.700 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:10.109 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:04:59.719 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:04:59.896 INFO:teuthology.orchestra.run.smithi094.stderr:no valid command found; 10 closest matches: 2024-08-30T16:04:59.897 INFO:teuthology.orchestra.run.smithi094.stderr:0 2024-08-30T16:04:59.897 INFO:teuthology.orchestra.run.smithi094.stderr:1 2024-08-30T16:04:59.897 INFO:teuthology.orchestra.run.smithi094.stderr:2 2024-08-30T16:04:59.897 INFO:teuthology.orchestra.run.smithi094.stderr:abort 2024-08-30T16:04:59.897 INFO:teuthology.orchestra.run.smithi094.stderr:assert 2024-08-30T16:04:59.897 INFO:teuthology.orchestra.run.smithi094.stderr:config diff 2024-08-30T16:04:59.897 INFO:teuthology.orchestra.run.smithi094.stderr:config diff get 2024-08-30T16:04:59.897 INFO:teuthology.orchestra.run.smithi094.stderr:config get 2024-08-30T16:04:59.897 INFO:teuthology.orchestra.run.smithi094.stderr:config help [] 2024-08-30T16:04:59.897 INFO:teuthology.orchestra.run.smithi094.stderr:config set ... 2024-08-30T16:04:59.897 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: invalid command 2024-08-30T16:04:59.899 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T16:04:59.899 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-08-30T16:04:59.988 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:10.397 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:05:00.523 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:10.932 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:05:00.553 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:10.962 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:05:04.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:05:05.383 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:15.792 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:05:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:no valid command found; 10 closest matches: 2024-08-30T16:05:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:0 2024-08-30T16:05:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:1 2024-08-30T16:05:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:2 2024-08-30T16:05:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:abort 2024-08-30T16:05:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:assert 2024-08-30T16:05:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:bluefs debug_inject_read_zeros 2024-08-30T16:05:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:bluefs files list 2024-08-30T16:05:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:bluefs stats 2024-08-30T16:05:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:bluestore allocator dump block 2024-08-30T16:05:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:bluestore allocator fragmentation block 2024-08-30T16:05:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: invalid command 2024-08-30T16:05:05.403 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T16:05:05.404 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-08-30T16:05:10.300 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:20.708 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:05:10.304 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:20.713 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:05:10.404 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:05:11.105 INFO:teuthology.orchestra.run.smithi094.stderr:no valid command found; 10 closest matches: 2024-08-30T16:05:11.105 INFO:teuthology.orchestra.run.smithi094.stderr:0 2024-08-30T16:05:11.105 INFO:teuthology.orchestra.run.smithi094.stderr:1 2024-08-30T16:05:11.106 INFO:teuthology.orchestra.run.smithi094.stderr:2 2024-08-30T16:05:11.106 INFO:teuthology.orchestra.run.smithi094.stderr:abort 2024-08-30T16:05:11.106 INFO:teuthology.orchestra.run.smithi094.stderr:assert 2024-08-30T16:05:11.106 INFO:teuthology.orchestra.run.smithi094.stderr:bluefs debug_inject_read_zeros 2024-08-30T16:05:11.106 INFO:teuthology.orchestra.run.smithi094.stderr:bluefs files list 2024-08-30T16:05:11.106 INFO:teuthology.orchestra.run.smithi094.stderr:bluefs stats 2024-08-30T16:05:11.106 INFO:teuthology.orchestra.run.smithi094.stderr:bluestore bluefs device info [] 2024-08-30T16:05:11.106 INFO:teuthology.orchestra.run.smithi094.stderr:config diff 2024-08-30T16:05:11.106 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: invalid command 2024-08-30T16:05:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T16:05:11.109 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-08-30T16:05:13.518 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:36.123 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:13.521 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:36.126 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:13.538 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:36.143 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:13.570 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:35.107 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:13.571 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:35.108 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:13.572 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:34.731 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:13.573 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:34.732 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:13.585 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:35.122 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:13.588 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:34.747 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:13.964 INFO:tasks.ceph.osd.6.smithi094.stderr:==00:00:00:24.372 45235== Warning: unimplemented fcntl command: 1038 2024-08-30T16:05:16.110 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:05:17.604 INFO:teuthology.orchestra.run.smithi094.stderr:no valid command found; 10 closest matches: 2024-08-30T16:05:17.604 INFO:teuthology.orchestra.run.smithi094.stderr:0 2024-08-30T16:05:17.604 INFO:teuthology.orchestra.run.smithi094.stderr:1 2024-08-30T16:05:17.604 INFO:teuthology.orchestra.run.smithi094.stderr:2 2024-08-30T16:05:17.604 INFO:teuthology.orchestra.run.smithi094.stderr:abort 2024-08-30T16:05:17.604 INFO:teuthology.orchestra.run.smithi094.stderr:assert 2024-08-30T16:05:17.604 INFO:teuthology.orchestra.run.smithi094.stderr:bluefs debug_inject_read_zeros 2024-08-30T16:05:17.604 INFO:teuthology.orchestra.run.smithi094.stderr:bluefs files list 2024-08-30T16:05:17.604 INFO:teuthology.orchestra.run.smithi094.stderr:bluefs stats 2024-08-30T16:05:17.604 INFO:teuthology.orchestra.run.smithi094.stderr:bluestore allocator dump block 2024-08-30T16:05:17.604 INFO:teuthology.orchestra.run.smithi094.stderr:bluestore allocator fragmentation block 2024-08-30T16:05:17.605 INFO:teuthology.orchestra.run.smithi094.stderr:admin_socket: invalid command 2024-08-30T16:05:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T16:05:17.607 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-08-30T16:05:20.784 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:05:20.775+0000 5ef1600 -1 osd.6 74 log_to_monitors true 2024-08-30T16:05:22.608 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-08-30T16:05:23.263 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-08-30T16:05:23.264 INFO:teuthology.orchestra.run.smithi094.stdout: "ops": [], 2024-08-30T16:05:23.264 INFO:teuthology.orchestra.run.smithi094.stdout: "num_ops": 0 2024-08-30T16:05:23.264 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-08-30T16:05:23.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-08-30T16:05:23.284 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:05:23.267+0000 14393640 -1 osd.6 104 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T16:05:24.265 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.0 to 1 (10000) 2024-08-30T16:05:24.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-08-30T16:05:25.258 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.1 to 1 (10000) 2024-08-30T16:05:25.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-08-30T16:05:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.3 to 1 (10000) 2024-08-30T16:05:26.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-08-30T16:05:27.480 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.7 to 1 (10000) 2024-08-30T16:05:27.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-08-30T16:05:28.507 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:51.112 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:28.554 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:50.091 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:28.556 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:50.093 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:28.560 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:51.165 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:28.577 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:50.114 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:28.582 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:51.187 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:28.611 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:49.771 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:28.614 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:49.774 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:28.633 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:49.793 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:28.794 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.4 to 1 (10000) 2024-08-30T16:05:28.811 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-08-30T16:05:28.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:05:29.361 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:05:29.362 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:05:29.373 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-08-30T16:05:28.688428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":795699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":138092,"kb_used_data":2980,"kb_used_omap":7,"kb_used_meta":135032,"kb_avail":471721108,"statfs":{"total":483183820800,"available":483042414592,"internally_reserved":0,"allocated":3051520,"data_stored":2101729,"data_compressed":1558,"data_compressed_allocated":49152,"data_compressed_original":786432,"omap_allocated":7967,"internal_metadata":138272993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.491964"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-08-30T16:04:57.587163+0000","last_change":"2024-08-30T16:04:11.379068+0000","last_active":"2024-08-30T16:04:57.587163+0000","last_peered":"2024-08-30T16:04:57.587163+0000","last_clean":"2024-08-30T16:04:57.587163+0000","last_became_active":"2024-08-30T16:04:11.370524+0000","last_became_peered":"2024-08-30T16:04:11.370524+0000","last_unstale":"2024-08-30T16:04:57.587163+0000","last_undegraded":"2024-08-30T16:04:57.587163+0000","last_fullsized":"2024-08-30T16:04:57.587163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:58:06.441243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-08-30T16:05:25.626438+0000","last_change":"2024-08-30T16:04:38.088583+0000","last_active":"2024-08-30T16:05:25.626438+0000","last_peered":"2024-08-30T16:05:25.626438+0000","last_clean":"2024-08-30T16:05:25.626438+0000","last_became_active":"2024-08-30T16:04:38.082821+0000","last_became_peered":"2024-08-30T16:04:38.082821+0000","last_unstale":"2024-08-30T16:05:25.626438+0000","last_undegraded":"2024-08-30T16:05:25.626438+0000","last_fullsized":"2024-08-30T16:05:25.626438+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:40:53.077427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":107,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-08-30T16:04:58.081759+0000","last_change":"2024-08-30T16:04:37.711140+0000","last_active":"2024-08-30T16:04:58.081759+0000","last_peered":"2024-08-30T16:04:58.081759+0000","last_clean":"2024-08-30T16:04:58.081759+0000","last_became_active":"2024-08-30T16:04:37.703814+0000","last_became_peered":"2024-08-30T16:04:37.703814+0000","last_unstale":"2024-08-30T16:04:58.081759+0000","last_undegraded":"2024-08-30T16:04:58.081759+0000","last_fullsized":"2024-08-30T16:04:58.081759+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T02:31:37.780701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":92,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-08-30T16:04:57.588837+0000","last_change":"2024-08-30T16:04:37.773968+0000","last_active":"2024-08-30T16:04:57.588837+0000","last_peered":"2024-08-30T16:04:57.588837+0000","last_clean":"2024-08-30T16:04:57.588837+0000","last_became_active":"2024-08-30T16:04:37.764408+0000","last_became_peered":"2024-08-30T16:04:37.764408+0000","last_unstale":"2024-08-30T16:04:57.588837+0000","last_undegraded":"2024-08-30T16:04:57.588837+0000","last_fullsized":"2024-08-30T16:04:57.588837+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T00:16:11.648089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":105,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-08-30T16:04:58.085042+0000","last_change":"2024-08-30T16:01:01.435839+0000","last_active":"2024-08-30T16:04:58.085042+0000","last_peered":"2024-08-30T16:04:58.085042+0000","last_clean":"2024-08-30T16:04:58.085042+0000","last_became_active":"2024-08-30T16:01:00.974833+0000","last_became_peered":"2024-08-30T16:01:00.974833+0000","last_unstale":"2024-08-30T16:04:58.085042+0000","last_undegraded":"2024-08-30T16:04:58.085042+0000","last_fullsized":"2024-08-30T16:04:58.085042+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:09:38.575833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":105,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-08-30T16:04:57.582741+0000","last_change":"2024-08-30T16:04:37.756175+0000","last_active":"2024-08-30T16:04:57.582741+0000","last_peered":"2024-08-30T16:04:57.582741+0000","last_clean":"2024-08-30T16:04:57.582741+0000","last_became_active":"2024-08-30T16:04:37.750394+0000","last_became_peered":"2024-08-30T16:04:37.750394+0000","last_unstale":"2024-08-30T16:04:57.582741+0000","last_undegraded":"2024-08-30T16:04:57.582741+0000","last_fullsized":"2024-08-30T16:04:57.582741+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:00:53.987924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":108,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-08-30T16:05:23.527040+0000","last_change":"2024-08-30T16:04:37.685351+0000","last_active":"2024-08-30T16:05:23.527040+0000","last_peered":"2024-08-30T16:05:23.527040+0000","last_clean":"2024-08-30T16:05:23.527040+0000","last_became_active":"2024-08-30T16:04:37.678764+0000","last_became_peered":"2024-08-30T16:04:37.678764+0000","last_unstale":"2024-08-30T16:05:23.527040+0000","last_undegraded":"2024-08-30T16:05:23.527040+0000","last_fullsized":"2024-08-30T16:05:23.527040+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T02:39:43.852577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":35,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-08-30T16:04:57.585441+0000","last_change":"2024-08-30T16:04:08.039936+0000","last_active":"2024-08-30T16:04:57.585441+0000","last_peered":"2024-08-30T16:04:57.585441+0000","last_clean":"2024-08-30T16:04:57.585441+0000","last_became_active":"2024-08-30T16:04:07.760364+0000","last_became_peered":"2024-08-30T16:04:07.760364+0000","last_unstale":"2024-08-30T16:04:57.585441+0000","last_undegraded":"2024-08-30T16:04:57.585441+0000","last_fullsized":"2024-08-30T16:04:57.585441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:13:37.016725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":109,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-08-30T16:05:23.520040+0000","last_change":"2024-08-30T16:04:27.064448+0000","last_active":"2024-08-30T16:05:23.520040+0000","last_peered":"2024-08-30T16:05:23.520040+0000","last_clean":"2024-08-30T16:05:23.520040+0000","last_became_active":"2024-08-30T16:04:26.684625+0000","last_became_peered":"2024-08-30T16:04:26.684625+0000","last_unstale":"2024-08-30T16:05:23.520040+0000","last_undegraded":"2024-08-30T16:05:23.520040+0000","last_fullsized":"2024-08-30T16:05:23.520040+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:46:40.994708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":622592,"data_stored":655458,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":13.491,"5min":12.247,"15min":13.696},"min":{"1min":5.0910000000000002,"5min":4.8049999999999997,"15min":4.8049999999999997},"max":{"1min":45.085000000000001,"5min":57.177999999999997,"15min":57.177999999999997},"last":5.5519999999999996},{"interface":"front","average":{"1min":11.670999999999999,"5min":12.109999999999999,"15min":12.93},"min":{"1min":4.9109999999999996,"5min":4.9109999999999996,"15min":4.9109999999999996},"max":{"1min":33.634999999999998,"5min":58.024999999999999,"15min":58.024999999999999},"last":8.7620000000000005}]},{"osd":2,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":11.814,"5min":11.039,"15min":10.632},"min":{"1min":6.4050000000000002,"5min":5.5780000000000003,"15min":5.5780000000000003},"max":{"1min":45.244,"5min":73.399000000000001,"15min":73.399000000000001},"last":11.803000000000001},{"interface":"front","average":{"1min":15.143000000000001,"5min":14.789999999999999,"15min":14.981999999999999},"min":{"1min":6.0049999999999999,"5min":6.0049999999999999,"15min":6.0049999999999999},"max":{"1min":43.427,"5min":75.120000000000005,"15min":75.120000000000005},"last":6.5640000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":15.519,"5min":16.143000000000001,"15min":17.157},"min":{"1min":9.7870000000000008,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":29.106999999999999,"5min":66.869,"15min":66.869},"last":13.058999999999999},{"interface":"front","average":{"1min":12.119999999999999,"5min":13.970000000000001,"15min":14.608000000000001},"min":{"1min":7.6520000000000001,"5min":7.6520000000000001,"15min":7.6520000000000001},"max":{"1min":23.492000000000001,"5min":104.38200000000001,"15min":104.38200000000001},"last":9.9900000000000002}]},{"osd":4,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":14.872999999999999,"5min":15.234,"15min":14.93},"min":{"1min":7.3470000000000004,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":26.141999999999999,"5min":66.340999999999994,"15min":66.340999999999994},"last":10.523},{"interface":"front","average":{"1min":13.529,"5min":13.217000000000001,"15min":13.115},"min":{"1min":7.0119999999999996,"5min":7.0039999999999996,"15min":7.0039999999999996},"max":{"1min":27.724,"5min":29.818999999999999,"15min":29.818999999999999},"last":11.193}]},{"osd":5,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":17.216999999999999,"5min":16.297000000000001,"15min":16.709},"min":{"1min":8.9320000000000004,"5min":7.8920000000000003,"15min":7.8920000000000003},"max":{"1min":51.021999999999998,"5min":55.784999999999997,"15min":55.784999999999997},"last":7.21},{"interface":"front","average":{"1min":19.859000000000002,"5min":18.872,"15min":19.481000000000002},"min":{"1min":10.504,"5min":9.5800000000000001,"15min":9.5800000000000001},"max":{"1min":52.845999999999997,"5min":68.004999999999995,"15min":68.004999999999995},"last":12.367000000000001}]},{"osd":6,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":15.708,"5min":16.609000000000002,"15min":17.382000000000001},"min":{"1min":10.180999999999999,"5min":9.1929999999999996,"15min":9.1929999999999996},"max":{"1min":27.353999999999999,"5min":58.201999999999998,"15min":58.201999999999998},"last":18.898},{"interface":"front","average":{"1min":16.324999999999999,"5min":16.748000000000001,"15min":17.001000000000001},"min":{"1min":9.6980000000000004,"5min":8.8390000000000004,"15min":8.8390000000000004},"max":{"1min":26.257999999999999,"5min":37.253,"15min":37.253},"last":12.787000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":17.847999999999999,"5min":18.407,"15min":19.649000000000001},"min":{"1min":11.308,"5min":8.2050000000000001,"15min":8.2050000000000001},"max":{"1min":33.444000000000003,"5min":68.950000000000003,"15min":68.950000000000003},"last":15.055999999999999},{"interface":"front","average":{"1min":16.82,"5min":17.565000000000001,"15min":18.876999999999999},"min":{"1min":8.5660000000000007,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":36.186,"5min":59.079000000000001,"15min":59.079000000000001},"last":9.4529999999999994}]}]},{"osd":7,"up_from":21,"seq":90194313297,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":618496,"data_stored":655439,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":6.9130000000000003,"5min":8.8819999999999997,"15min":8.9359999999999999},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":11.773999999999999,"5min":30.600999999999999,"15min":30.600999999999999},"last":4.3789999999999996},{"interface":"front","average":{"1min":8.4130000000000003,"5min":9.2750000000000004,"15min":9.173},"min":{"1min":4.2590000000000003,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":19.460999999999999,"5min":28.077000000000002,"15min":28.077000000000002},"last":5.5780000000000003}]},{"osd":1,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":11.005000000000001,"5min":12.624000000000001,"15min":11.432},"min":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"max":{"1min":40.799999999999997,"5min":83.581000000000003,"15min":83.581000000000003},"last":18.818000000000001},{"interface":"front","average":{"1min":11.51,"5min":12.407999999999999,"15min":12.407},"min":{"1min":5.7439999999999998,"5min":5.7439999999999998,"15min":5.7439999999999998},"max":{"1min":38.768999999999998,"5min":81.722999999999999,"15min":81.722999999999999},"last":19.524999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":12.596,"5min":13.641999999999999,"15min":13.606},"min":{"1min":5.4850000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"max":{"1min":32.988999999999997,"5min":42.506,"15min":42.506},"last":25.885999999999999},{"interface":"front","average":{"1min":13.048,"5min":14.539,"15min":16.524999999999999},"min":{"1min":8.0050000000000008,"5min":6.9800000000000004,"15min":6.9800000000000004},"max":{"1min":24.622,"5min":43.484999999999999,"15min":43.484999999999999},"last":21.966000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":14.619,"5min":16.202999999999999,"15min":16.465},"min":{"1min":9.0909999999999993,"5min":8.8420000000000005,"15min":8.8420000000000005},"max":{"1min":26.826000000000001,"5min":65.739999999999995,"15min":65.739999999999995},"last":21.427},{"interface":"front","average":{"1min":13.359999999999999,"5min":13.337,"15min":12.888999999999999},"min":{"1min":6.4690000000000003,"5min":6.2119999999999997,"15min":6.2119999999999997},"max":{"1min":28.779,"5min":64.980000000000004,"15min":64.980000000000004},"last":16.308}]},{"osd":4,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.955,"15min":12.871},"min":{"1min":7.1349999999999998,"5min":7.0339999999999998,"15min":7.0339999999999998},"max":{"1min":41.631,"5min":41.631,"15min":41.631},"last":23.213999999999999},{"interface":"front","average":{"1min":16.166,"5min":16.073,"15min":17.216999999999999},"min":{"1min":8.8729999999999993,"5min":8.8729999999999993,"15min":8.8729999999999993},"max":{"1min":29.516999999999999,"5min":37.393000000000001,"15min":37.393000000000001},"last":25.149999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":11.723000000000001,"5min":11.907,"15min":11.522},"min":{"1min":6.2380000000000004,"5min":6.2309999999999999,"15min":6.2309999999999999},"max":{"1min":33.534999999999997,"5min":33.534999999999997,"15min":33.534999999999997},"last":17.488},{"interface":"front","average":{"1min":14.628,"5min":15.861000000000001,"15min":16.631},"min":{"1min":8.2870000000000008,"5min":8.2870000000000008,"15min":8.2870000000000008},"max":{"1min":31.853999999999999,"5min":38.618000000000002,"15min":38.618000000000002},"last":23.952999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.326000000000001,"5min":13.771000000000001,"15min":13.208},"min":{"1min":8.0060000000000002,"5min":6.3230000000000004,"15min":6.3230000000000004},"max":{"1min":19.231000000000002,"5min":42.829999999999998,"15min":42.829999999999998},"last":11.368},{"interface":"front","average":{"1min":15.842000000000001,"5min":17.606000000000002,"15min":18.088999999999999},"min":{"1min":11.712,"5min":10.749000000000001,"15min":10.749000000000001},"max":{"1min":22.695,"5min":44.491999999999997,"15min":44.491999999999997},"last":14.314}]}]},{"osd":6,"up_from":0,"seq":450971566083,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":21,"seq":90194313297,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27596,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344244,"statfs":{"total":96636764160,"available":96608505856,"internally_reserved":0,"allocated":589824,"data_stored":257618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":10.009,"5min":9.9629999999999992,"15min":11.073},"min":{"1min":4.3810000000000002,"5min":4.29,"15min":4.29},"max":{"1min":46.146999999999998,"5min":50.500999999999998,"15min":50.500999999999998},"last":15.329000000000001},{"interface":"front","average":{"1min":9.9789999999999992,"5min":11.208,"15min":13.714},"min":{"1min":5.3890000000000002,"5min":5.1360000000000001,"15min":5.1360000000000001},"max":{"1min":18.420999999999999,"5min":51.965000000000003,"15min":51.965000000000003},"last":8.6440000000000001}]},{"osd":1,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":15.244,"5min":15.003,"15min":18.209},"min":{"1min":7.6230000000000002,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":44.862000000000002,"5min":137.56700000000001,"15min":137.56700000000001},"last":7.7549999999999999},{"interface":"front","average":{"1min":12.853,"5min":13.709,"15min":16.225999999999999},"min":{"1min":5.5720000000000001,"5min":5.0060000000000002,"15min":5.0060000000000002},"max":{"1min":47.530999999999999,"5min":136.321,"15min":136.321},"last":12.265000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":12.148,"5min":13.746,"15min":15.852},"min":{"1min":6.3470000000000004,"5min":6.3470000000000004,"15min":6.3470000000000004},"max":{"1min":28.602,"5min":73.831000000000003,"15min":73.831000000000003},"last":12.994},{"interface":"front","average":{"1min":12.590999999999999,"5min":12.757,"15min":15.281000000000001},"min":{"1min":6.2830000000000004,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":26.074999999999999,"5min":71.831999999999994,"15min":71.831999999999994},"last":10.042}]},{"osd":4,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":13.956,"5min":18.161999999999999,"15min":27.286999999999999},"min":{"1min":6.9199999999999999,"5min":6.9199999999999999,"15min":6.9199999999999999},"max":{"1min":27.263999999999999,"5min":342.34500000000003,"15min":342.34500000000003},"last":14.771000000000001},{"interface":"front","average":{"1min":14.484,"5min":16.646999999999998,"15min":19.646999999999998},"min":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"max":{"1min":33.375999999999998,"5min":127.857,"15min":127.857},"last":11.488}]},{"osd":5,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":13.752000000000001,"5min":15.323,"15min":15.843},"min":{"1min":6.5,"5min":6.5,"15min":6.5},"max":{"1min":37.534999999999997,"5min":58.715000000000003,"15min":58.715000000000003},"last":13.574999999999999},{"interface":"front","average":{"1min":15.122,"5min":17.899000000000001,"15min":18.965},"min":{"1min":8.5310000000000006,"5min":7.2370000000000001,"15min":7.2370000000000001},"max":{"1min":38.450000000000003,"5min":58.212000000000003,"15min":58.212000000000003},"last":10.592000000000001}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.295000000000002,"5min":23.492000000000001,"15min":30.053999999999998},"min":{"1min":11.096,"5min":8.6630000000000003,"15min":8.6630000000000003},"max":{"1min":25.298999999999999,"5min":388.96100000000001,"15min":388.96100000000001},"last":15.696},{"interface":"front","average":{"1min":13.961,"5min":24.18,"15min":32.835999999999999},"min":{"1min":9.4079999999999995,"5min":8.5519999999999996,"15min":8.5519999999999996},"max":{"1min":20.439,"5min":448.69200000000001,"15min":448.69200000000001},"last":13.44}]},{"osd":7,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":22.033000000000001,"5min":20.844999999999999,"15min":25.013000000000002},"min":{"1min":7.3049999999999997,"5min":7.3049999999999997,"15min":7.3049999999999997},"max":{"1min":91.646000000000001,"5min":177.596,"15min":177.596},"last":14.180999999999999},{"interface":"front","average":{"1min":21.826000000000001,"5min":21.407,"15min":26.369},"min":{"1min":7.476,"5min":7.476,"15min":7.476},"max":{"1min":101.158,"5min":231.03800000000001,"15min":231.03800000000001},"last":6.891}]}]},{"osd":4,"up_from":20,"seq":85899346001,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":610304,"data_stored":266607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":8.0920000000000005,"5min":8.8539999999999992,"15min":11.156000000000001},"min":{"1min":5.0110000000000001,"5min":4.5439999999999996,"15min":4.5439999999999996},"max":{"1min":11.35,"5min":35.052,"15min":35.817999999999998},"last":5.0060000000000002},{"interface":"front","average":{"1min":7.8849999999999998,"5min":9.2420000000000009,"15min":11.247},"min":{"1min":4.6500000000000004,"5min":4.6500000000000004,"15min":4.6500000000000004},"max":{"1min":16.132999999999999,"5min":35.091999999999999,"15min":35.091999999999999},"last":7.7000000000000002}]},{"osd":1,"last update":"Fri Aug 30 16:04:28 2024","interfaces":[{"interface":"back","average":{"1min":12.51,"5min":12.608000000000001,"15min":11.776},"min":{"1min":5.6879999999999997,"5min":4.7000000000000002,"15min":4.7000000000000002},"max":{"1min":27.940000000000001,"5min":34.347999999999999,"15min":34.347999999999999},"last":20.111000000000001},{"interface":"front","average":{"1min":10.170999999999999,"5min":12.247,"15min":12.433999999999999},"min":{"1min":7.3769999999999998,"5min":6.5590000000000002,"15min":6.5590000000000002},"max":{"1min":14.595000000000001,"5min":35.481000000000002,"15min":35.481000000000002},"last":8.9090000000000007}]},{"osd":2,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":9.6660000000000004,"5min":9.5470000000000006,"15min":9.9280000000000008},"min":{"1min":5.4930000000000003,"5min":5.4279999999999999,"15min":5.4279999999999999},"max":{"1min":30.666,"5min":30.666,"15min":30.666},"last":16.872},{"interface":"front","average":{"1min":11.707000000000001,"5min":12.987,"15min":14.148999999999999},"min":{"1min":7.0629999999999997,"5min":7.0629999999999997,"15min":7.0629999999999997},"max":{"1min":21.355,"5min":29.079999999999998,"15min":30.629000000000001},"last":15.125}]},{"osd":3,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":11.263,"5min":11.942,"15min":12.234999999999999},"min":{"1min":7.476,"5min":5.8010000000000002,"15min":5.8010000000000002},"max":{"1min":28.315999999999999,"5min":29.715,"15min":29.715},"last":7.1260000000000003},{"interface":"front","average":{"1min":13.385,"5min":14.148999999999999,"15min":15.195},"min":{"1min":7.8220000000000001,"5min":7.5830000000000002,"15min":7.5830000000000002},"max":{"1min":27.321999999999999,"5min":27.321999999999999,"15min":31.73},"last":15.651}]},{"osd":5,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":14.595000000000001,"5min":13.505000000000001,"15min":11.943},"min":{"1min":9.1199999999999992,"5min":6.8639999999999999,"15min":6.5010000000000003},"max":{"1min":31.565000000000001,"5min":42.143000000000001,"15min":42.143000000000001},"last":18.326000000000001},{"interface":"front","average":{"1min":12.859,"5min":13.661,"15min":14.194000000000001},"min":{"1min":6.5579999999999998,"5min":6.5579999999999998,"15min":6.5579999999999998},"max":{"1min":29.210999999999999,"5min":43.917000000000002,"15min":43.917000000000002},"last":21.623999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:23 2024","interfaces":[{"interface":"back","average":{"1min":13.481,"5min":14.797000000000001,"15min":15.696999999999999},"min":{"1min":8.0359999999999996,"5min":6.5289999999999999,"15min":6.5289999999999999},"max":{"1min":20.704000000000001,"5min":37.215000000000003,"15min":37.215000000000003},"last":17.648},{"interface":"front","average":{"1min":16.030999999999999,"5min":16.199000000000002,"15min":17.021999999999998},"min":{"1min":9.8300000000000001,"5min":8.7870000000000008,"15min":8.7870000000000008},"max":{"1min":24.986999999999998,"5min":37.753999999999998,"15min":37.753999999999998},"last":23.215}]},{"osd":7,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":14.657,"15min":15.507999999999999},"min":{"1min":7.9950000000000001,"5min":7.7130000000000001,"15min":7.7130000000000001},"max":{"1min":21.992999999999999,"5min":33.314,"15min":34.551000000000002},"last":27.103000000000002},{"interface":"front","average":{"1min":12.443,"5min":15.465,"15min":16.873999999999999},"min":{"1min":6.375,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":18.116,"5min":35.701999999999998,"15min":35.701999999999998},"last":21.116}]}]},{"osd":2,"up_from":0,"seq":68719476815,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":15,"seq":64424509518,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":266607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:04:33 2024","interfaces":[{"interface":"back","average":{"1min":10.163,"5min":10.477,"15min":9.9339999999999993},"min":{"1min":4.7489999999999997,"5min":4.7489999999999997,"15min":4.7489999999999997},"max":{"1min":40.735999999999997,"5min":40.735999999999997,"15min":40.735999999999997},"last":5.8940000000000001},{"interface":"front","average":{"1min":12.625999999999999,"5min":11.563000000000001,"15min":11.423},"min":{"1min":4.383,"5min":4.383,"15min":4.383},"max":{"1min":42.387,"5min":42.387,"15min":42.387},"last":5.0069999999999997}]},{"osd":2,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":10.721,"5min":11.506,"15min":11.308},"min":{"1min":6.3899999999999997,"5min":5.3689999999999998,"15min":3.3370000000000002},"max":{"1min":24.170999999999999,"5min":46.768999999999998,"15min":46.768999999999998},"last":12.143000000000001},{"interface":"front","average":{"1min":13.676,"5min":13.199999999999999,"15min":14.022},"min":{"1min":6.8860000000000001,"5min":6.2249999999999996,"15min":5.702},"max":{"1min":31.361000000000001,"5min":43.921999999999997,"15min":43.921999999999997},"last":9.6780000000000008}]},{"osd":3,"last update":"Fri Aug 30 16:04:34 2024","interfaces":[{"interface":"back","average":{"1min":13.814,"5min":16.356000000000002,"15min":22.327999999999999},"min":{"1min":6.6680000000000001,"5min":6.2839999999999998,"15min":6.2839999999999998},"max":{"1min":38.140999999999998,"5min":200.155,"15min":200.155},"last":10.26},{"interface":"front","average":{"1min":15.974,"5min":20.137,"15min":27.510999999999999},"min":{"1min":8.4339999999999993,"5min":7.0940000000000003,"15min":7.0940000000000003},"max":{"1min":35.576999999999998,"5min":267.03100000000001,"15min":267.03100000000001},"last":23.777999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:04:34 2024","interfaces":[{"interface":"back","average":{"1min":11.683,"5min":14.086,"15min":15.364000000000001},"min":{"1min":6.0970000000000004,"5min":6.0970000000000004,"15min":6.0970000000000004},"max":{"1min":23.608000000000001,"5min":56.200000000000003,"15min":56.200000000000003},"last":8.625},{"interface":"front","average":{"1min":15.760999999999999,"5min":16.109000000000002,"15min":15.863},"min":{"1min":9.298,"5min":8.609,"15min":8.609},"max":{"1min":24.710000000000001,"5min":48.534999999999997,"15min":48.534999999999997},"last":11.393000000000001}]},{"osd":5,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":14.43,"5min":15.682,"15min":16.082999999999998},"min":{"1min":9.8059999999999992,"5min":5.9199999999999999,"15min":5.6920000000000002},"max":{"1min":40.006,"5min":43.308,"15min":43.308},"last":24.443999999999999},{"interface":"front","average":{"1min":16.951000000000001,"5min":17.693000000000001,"15min":17.853999999999999},"min":{"1min":9.9100000000000001,"5min":9.2149999999999999,"15min":4.75},"max":{"1min":49.575000000000003,"5min":49.575000000000003,"15min":49.575000000000003},"last":12.695}]},{"osd":6,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":13.808999999999999,"5min":14.292,"15min":14.385999999999999},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":39.460999999999999,"5min":59.328000000000003,"15min":59.328000000000003},"last":23.652999999999999},{"interface":"front","average":{"1min":19.225000000000001,"5min":18.331,"15min":18.533999999999999},"min":{"1min":10.781000000000001,"5min":10.005000000000001,"15min":10.005000000000001},"max":{"1min":46.237000000000002,"5min":58.567,"15min":58.567},"last":38.289000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:04:33 2024","interfaces":[{"interface":"back","average":{"1min":18.983000000000001,"5min":19.995000000000001,"15min":21.375},"min":{"1min":12.195,"5min":9.1180000000000003,"15min":9.1180000000000003},"max":{"1min":34.927999999999997,"5min":59.515999999999998,"15min":59.515999999999998},"last":13.212999999999999},{"interface":"front","average":{"1min":12.894,"5min":14.285,"15min":15.276999999999999},"min":{"1min":7.6379999999999999,"5min":7.1619999999999999,"15min":7.1619999999999999},"max":{"1min":24.966000000000001,"5min":58.807000000000002,"15min":58.807000000000002},"last":8.3520000000000003}]}]},{"osd":5,"up_from":0,"seq":55834574926,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:05:29.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:05:29.931 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:05:29.932 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:05:29.944 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-08-30T16:05:28.688428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":795699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":138092,"kb_used_data":2980,"kb_used_omap":7,"kb_used_meta":135032,"kb_avail":471721108,"statfs":{"total":483183820800,"available":483042414592,"internally_reserved":0,"allocated":3051520,"data_stored":2101729,"data_compressed":1558,"data_compressed_allocated":49152,"data_compressed_original":786432,"omap_allocated":7967,"internal_metadata":138272993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.491964"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-08-30T16:04:57.587163+0000","last_change":"2024-08-30T16:04:11.379068+0000","last_active":"2024-08-30T16:04:57.587163+0000","last_peered":"2024-08-30T16:04:57.587163+0000","last_clean":"2024-08-30T16:04:57.587163+0000","last_became_active":"2024-08-30T16:04:11.370524+0000","last_became_peered":"2024-08-30T16:04:11.370524+0000","last_unstale":"2024-08-30T16:04:57.587163+0000","last_undegraded":"2024-08-30T16:04:57.587163+0000","last_fullsized":"2024-08-30T16:04:57.587163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:58:06.441243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-08-30T16:05:25.626438+0000","last_change":"2024-08-30T16:04:38.088583+0000","last_active":"2024-08-30T16:05:25.626438+0000","last_peered":"2024-08-30T16:05:25.626438+0000","last_clean":"2024-08-30T16:05:25.626438+0000","last_became_active":"2024-08-30T16:04:38.082821+0000","last_became_peered":"2024-08-30T16:04:38.082821+0000","last_unstale":"2024-08-30T16:05:25.626438+0000","last_undegraded":"2024-08-30T16:05:25.626438+0000","last_fullsized":"2024-08-30T16:05:25.626438+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:40:53.077427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":107,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-08-30T16:04:58.081759+0000","last_change":"2024-08-30T16:04:37.711140+0000","last_active":"2024-08-30T16:04:58.081759+0000","last_peered":"2024-08-30T16:04:58.081759+0000","last_clean":"2024-08-30T16:04:58.081759+0000","last_became_active":"2024-08-30T16:04:37.703814+0000","last_became_peered":"2024-08-30T16:04:37.703814+0000","last_unstale":"2024-08-30T16:04:58.081759+0000","last_undegraded":"2024-08-30T16:04:58.081759+0000","last_fullsized":"2024-08-30T16:04:58.081759+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T02:31:37.780701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":92,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-08-30T16:04:57.588837+0000","last_change":"2024-08-30T16:04:37.773968+0000","last_active":"2024-08-30T16:04:57.588837+0000","last_peered":"2024-08-30T16:04:57.588837+0000","last_clean":"2024-08-30T16:04:57.588837+0000","last_became_active":"2024-08-30T16:04:37.764408+0000","last_became_peered":"2024-08-30T16:04:37.764408+0000","last_unstale":"2024-08-30T16:04:57.588837+0000","last_undegraded":"2024-08-30T16:04:57.588837+0000","last_fullsized":"2024-08-30T16:04:57.588837+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T00:16:11.648089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":105,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-08-30T16:04:58.085042+0000","last_change":"2024-08-30T16:01:01.435839+0000","last_active":"2024-08-30T16:04:58.085042+0000","last_peered":"2024-08-30T16:04:58.085042+0000","last_clean":"2024-08-30T16:04:58.085042+0000","last_became_active":"2024-08-30T16:01:00.974833+0000","last_became_peered":"2024-08-30T16:01:00.974833+0000","last_unstale":"2024-08-30T16:04:58.085042+0000","last_undegraded":"2024-08-30T16:04:58.085042+0000","last_fullsized":"2024-08-30T16:04:58.085042+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:09:38.575833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":105,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-08-30T16:04:57.582741+0000","last_change":"2024-08-30T16:04:37.756175+0000","last_active":"2024-08-30T16:04:57.582741+0000","last_peered":"2024-08-30T16:04:57.582741+0000","last_clean":"2024-08-30T16:04:57.582741+0000","last_became_active":"2024-08-30T16:04:37.750394+0000","last_became_peered":"2024-08-30T16:04:37.750394+0000","last_unstale":"2024-08-30T16:04:57.582741+0000","last_undegraded":"2024-08-30T16:04:57.582741+0000","last_fullsized":"2024-08-30T16:04:57.582741+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:00:53.987924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":108,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-08-30T16:05:23.527040+0000","last_change":"2024-08-30T16:04:37.685351+0000","last_active":"2024-08-30T16:05:23.527040+0000","last_peered":"2024-08-30T16:05:23.527040+0000","last_clean":"2024-08-30T16:05:23.527040+0000","last_became_active":"2024-08-30T16:04:37.678764+0000","last_became_peered":"2024-08-30T16:04:37.678764+0000","last_unstale":"2024-08-30T16:05:23.527040+0000","last_undegraded":"2024-08-30T16:05:23.527040+0000","last_fullsized":"2024-08-30T16:05:23.527040+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T02:39:43.852577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":35,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-08-30T16:04:57.585441+0000","last_change":"2024-08-30T16:04:08.039936+0000","last_active":"2024-08-30T16:04:57.585441+0000","last_peered":"2024-08-30T16:04:57.585441+0000","last_clean":"2024-08-30T16:04:57.585441+0000","last_became_active":"2024-08-30T16:04:07.760364+0000","last_became_peered":"2024-08-30T16:04:07.760364+0000","last_unstale":"2024-08-30T16:04:57.585441+0000","last_undegraded":"2024-08-30T16:04:57.585441+0000","last_fullsized":"2024-08-30T16:04:57.585441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:13:37.016725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":109,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-08-30T16:05:23.520040+0000","last_change":"2024-08-30T16:04:27.064448+0000","last_active":"2024-08-30T16:05:23.520040+0000","last_peered":"2024-08-30T16:05:23.520040+0000","last_clean":"2024-08-30T16:05:23.520040+0000","last_became_active":"2024-08-30T16:04:26.684625+0000","last_became_peered":"2024-08-30T16:04:26.684625+0000","last_unstale":"2024-08-30T16:05:23.520040+0000","last_undegraded":"2024-08-30T16:05:23.520040+0000","last_fullsized":"2024-08-30T16:05:23.520040+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:46:40.994708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":1193520,"data_compressed":2247,"data_compressed_allocated":73728,"data_compressed_original":1175552,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":622592,"data_stored":655458,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":13.491,"5min":12.247,"15min":13.696},"min":{"1min":5.0910000000000002,"5min":4.8049999999999997,"15min":4.8049999999999997},"max":{"1min":45.085000000000001,"5min":57.177999999999997,"15min":57.177999999999997},"last":5.5519999999999996},{"interface":"front","average":{"1min":11.670999999999999,"5min":12.109999999999999,"15min":12.93},"min":{"1min":4.9109999999999996,"5min":4.9109999999999996,"15min":4.9109999999999996},"max":{"1min":33.634999999999998,"5min":58.024999999999999,"15min":58.024999999999999},"last":8.7620000000000005}]},{"osd":2,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":11.814,"5min":11.039,"15min":10.632},"min":{"1min":6.4050000000000002,"5min":5.5780000000000003,"15min":5.5780000000000003},"max":{"1min":45.244,"5min":73.399000000000001,"15min":73.399000000000001},"last":11.803000000000001},{"interface":"front","average":{"1min":15.143000000000001,"5min":14.789999999999999,"15min":14.981999999999999},"min":{"1min":6.0049999999999999,"5min":6.0049999999999999,"15min":6.0049999999999999},"max":{"1min":43.427,"5min":75.120000000000005,"15min":75.120000000000005},"last":6.5640000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":15.519,"5min":16.143000000000001,"15min":17.157},"min":{"1min":9.7870000000000008,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":29.106999999999999,"5min":66.869,"15min":66.869},"last":13.058999999999999},{"interface":"front","average":{"1min":12.119999999999999,"5min":13.970000000000001,"15min":14.608000000000001},"min":{"1min":7.6520000000000001,"5min":7.6520000000000001,"15min":7.6520000000000001},"max":{"1min":23.492000000000001,"5min":104.38200000000001,"15min":104.38200000000001},"last":9.9900000000000002}]},{"osd":4,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":14.872999999999999,"5min":15.234,"15min":14.93},"min":{"1min":7.3470000000000004,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":26.141999999999999,"5min":66.340999999999994,"15min":66.340999999999994},"last":10.523},{"interface":"front","average":{"1min":13.529,"5min":13.217000000000001,"15min":13.115},"min":{"1min":7.0119999999999996,"5min":7.0039999999999996,"15min":7.0039999999999996},"max":{"1min":27.724,"5min":29.818999999999999,"15min":29.818999999999999},"last":11.193}]},{"osd":5,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":17.216999999999999,"5min":16.297000000000001,"15min":16.709},"min":{"1min":8.9320000000000004,"5min":7.8920000000000003,"15min":7.8920000000000003},"max":{"1min":51.021999999999998,"5min":55.784999999999997,"15min":55.784999999999997},"last":7.21},{"interface":"front","average":{"1min":19.859000000000002,"5min":18.872,"15min":19.481000000000002},"min":{"1min":10.504,"5min":9.5800000000000001,"15min":9.5800000000000001},"max":{"1min":52.845999999999997,"5min":68.004999999999995,"15min":68.004999999999995},"last":12.367000000000001}]},{"osd":6,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":15.708,"5min":16.609000000000002,"15min":17.382000000000001},"min":{"1min":10.180999999999999,"5min":9.1929999999999996,"15min":9.1929999999999996},"max":{"1min":27.353999999999999,"5min":58.201999999999998,"15min":58.201999999999998},"last":18.898},{"interface":"front","average":{"1min":16.324999999999999,"5min":16.748000000000001,"15min":17.001000000000001},"min":{"1min":9.6980000000000004,"5min":8.8390000000000004,"15min":8.8390000000000004},"max":{"1min":26.257999999999999,"5min":37.253,"15min":37.253},"last":12.787000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:04:35 2024","interfaces":[{"interface":"back","average":{"1min":17.847999999999999,"5min":18.407,"15min":19.649000000000001},"min":{"1min":11.308,"5min":8.2050000000000001,"15min":8.2050000000000001},"max":{"1min":33.444000000000003,"5min":68.950000000000003,"15min":68.950000000000003},"last":15.055999999999999},{"interface":"front","average":{"1min":16.82,"5min":17.565000000000001,"15min":18.876999999999999},"min":{"1min":8.5660000000000007,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":36.186,"5min":59.079000000000001,"15min":59.079000000000001},"last":9.4529999999999994}]}]},{"osd":7,"up_from":21,"seq":90194313297,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":618496,"data_stored":655439,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":6.9130000000000003,"5min":8.8819999999999997,"15min":8.9359999999999999},"min":{"1min":4.3949999999999996,"5min":4.3949999999999996,"15min":4.3949999999999996},"max":{"1min":11.773999999999999,"5min":30.600999999999999,"15min":30.600999999999999},"last":4.3789999999999996},{"interface":"front","average":{"1min":8.4130000000000003,"5min":9.2750000000000004,"15min":9.173},"min":{"1min":4.2590000000000003,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":19.460999999999999,"5min":28.077000000000002,"15min":28.077000000000002},"last":5.5780000000000003}]},{"osd":1,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":11.005000000000001,"5min":12.624000000000001,"15min":11.432},"min":{"1min":5.7009999999999996,"5min":5.7009999999999996,"15min":5.7009999999999996},"max":{"1min":40.799999999999997,"5min":83.581000000000003,"15min":83.581000000000003},"last":18.818000000000001},{"interface":"front","average":{"1min":11.51,"5min":12.407999999999999,"15min":12.407},"min":{"1min":5.7439999999999998,"5min":5.7439999999999998,"15min":5.7439999999999998},"max":{"1min":38.768999999999998,"5min":81.722999999999999,"15min":81.722999999999999},"last":19.524999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":12.596,"5min":13.641999999999999,"15min":13.606},"min":{"1min":5.4850000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"max":{"1min":32.988999999999997,"5min":42.506,"15min":42.506},"last":25.885999999999999},{"interface":"front","average":{"1min":13.048,"5min":14.539,"15min":16.524999999999999},"min":{"1min":8.0050000000000008,"5min":6.9800000000000004,"15min":6.9800000000000004},"max":{"1min":24.622,"5min":43.484999999999999,"15min":43.484999999999999},"last":21.966000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":14.619,"5min":16.202999999999999,"15min":16.465},"min":{"1min":9.0909999999999993,"5min":8.8420000000000005,"15min":8.8420000000000005},"max":{"1min":26.826000000000001,"5min":65.739999999999995,"15min":65.739999999999995},"last":21.427},{"interface":"front","average":{"1min":13.359999999999999,"5min":13.337,"15min":12.888999999999999},"min":{"1min":6.4690000000000003,"5min":6.2119999999999997,"15min":6.2119999999999997},"max":{"1min":28.779,"5min":64.980000000000004,"15min":64.980000000000004},"last":16.308}]},{"osd":4,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":13.391,"5min":13.955,"15min":12.871},"min":{"1min":7.1349999999999998,"5min":7.0339999999999998,"15min":7.0339999999999998},"max":{"1min":41.631,"5min":41.631,"15min":41.631},"last":23.213999999999999},{"interface":"front","average":{"1min":16.166,"5min":16.073,"15min":17.216999999999999},"min":{"1min":8.8729999999999993,"5min":8.8729999999999993,"15min":8.8729999999999993},"max":{"1min":29.516999999999999,"5min":37.393000000000001,"15min":37.393000000000001},"last":25.149999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:04:25 2024","interfaces":[{"interface":"back","average":{"1min":11.723000000000001,"5min":11.907,"15min":11.522},"min":{"1min":6.2380000000000004,"5min":6.2309999999999999,"15min":6.2309999999999999},"max":{"1min":33.534999999999997,"5min":33.534999999999997,"15min":33.534999999999997},"last":17.488},{"interface":"front","average":{"1min":14.628,"5min":15.861000000000001,"15min":16.631},"min":{"1min":8.2870000000000008,"5min":8.2870000000000008,"15min":8.2870000000000008},"max":{"1min":31.853999999999999,"5min":38.618000000000002,"15min":38.618000000000002},"last":23.952999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.326000000000001,"5min":13.771000000000001,"15min":13.208},"min":{"1min":8.0060000000000002,"5min":6.3230000000000004,"15min":6.3230000000000004},"max":{"1min":19.231000000000002,"5min":42.829999999999998,"15min":42.829999999999998},"last":11.368},{"interface":"front","average":{"1min":15.842000000000001,"5min":17.606000000000002,"15min":18.088999999999999},"min":{"1min":11.712,"5min":10.749000000000001,"15min":10.749000000000001},"max":{"1min":22.695,"5min":44.491999999999997,"15min":44.491999999999997},"last":14.314}]}]},{"osd":6,"up_from":0,"seq":450971566083,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":21,"seq":90194313297,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27596,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344244,"statfs":{"total":96636764160,"available":96608505856,"internally_reserved":0,"allocated":589824,"data_stored":257618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":10.009,"5min":9.9629999999999992,"15min":11.073},"min":{"1min":4.3810000000000002,"5min":4.29,"15min":4.29},"max":{"1min":46.146999999999998,"5min":50.500999999999998,"15min":50.500999999999998},"last":15.329000000000001},{"interface":"front","average":{"1min":9.9789999999999992,"5min":11.208,"15min":13.714},"min":{"1min":5.3890000000000002,"5min":5.1360000000000001,"15min":5.1360000000000001},"max":{"1min":18.420999999999999,"5min":51.965000000000003,"15min":51.965000000000003},"last":8.6440000000000001}]},{"osd":1,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":15.244,"5min":15.003,"15min":18.209},"min":{"1min":7.6230000000000002,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":44.862000000000002,"5min":137.56700000000001,"15min":137.56700000000001},"last":7.7549999999999999},{"interface":"front","average":{"1min":12.853,"5min":13.709,"15min":16.225999999999999},"min":{"1min":5.5720000000000001,"5min":5.0060000000000002,"15min":5.0060000000000002},"max":{"1min":47.530999999999999,"5min":136.321,"15min":136.321},"last":12.265000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":12.148,"5min":13.746,"15min":15.852},"min":{"1min":6.3470000000000004,"5min":6.3470000000000004,"15min":6.3470000000000004},"max":{"1min":28.602,"5min":73.831000000000003,"15min":73.831000000000003},"last":12.994},{"interface":"front","average":{"1min":12.590999999999999,"5min":12.757,"15min":15.281000000000001},"min":{"1min":6.2830000000000004,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":26.074999999999999,"5min":71.831999999999994,"15min":71.831999999999994},"last":10.042}]},{"osd":4,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":13.956,"5min":18.161999999999999,"15min":27.286999999999999},"min":{"1min":6.9199999999999999,"5min":6.9199999999999999,"15min":6.9199999999999999},"max":{"1min":27.263999999999999,"5min":342.34500000000003,"15min":342.34500000000003},"last":14.771000000000001},{"interface":"front","average":{"1min":14.484,"5min":16.646999999999998,"15min":19.646999999999998},"min":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"max":{"1min":33.375999999999998,"5min":127.857,"15min":127.857},"last":11.488}]},{"osd":5,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":13.752000000000001,"5min":15.323,"15min":15.843},"min":{"1min":6.5,"5min":6.5,"15min":6.5},"max":{"1min":37.534999999999997,"5min":58.715000000000003,"15min":58.715000000000003},"last":13.574999999999999},{"interface":"front","average":{"1min":15.122,"5min":17.899000000000001,"15min":18.965},"min":{"1min":8.5310000000000006,"5min":7.2370000000000001,"15min":7.2370000000000001},"max":{"1min":38.450000000000003,"5min":58.212000000000003,"15min":58.212000000000003},"last":10.592000000000001}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.295000000000002,"5min":23.492000000000001,"15min":30.053999999999998},"min":{"1min":11.096,"5min":8.6630000000000003,"15min":8.6630000000000003},"max":{"1min":25.298999999999999,"5min":388.96100000000001,"15min":388.96100000000001},"last":15.696},{"interface":"front","average":{"1min":13.961,"5min":24.18,"15min":32.835999999999999},"min":{"1min":9.4079999999999995,"5min":8.5519999999999996,"15min":8.5519999999999996},"max":{"1min":20.439,"5min":448.69200000000001,"15min":448.69200000000001},"last":13.44}]},{"osd":7,"last update":"Fri Aug 30 16:04:26 2024","interfaces":[{"interface":"back","average":{"1min":22.033000000000001,"5min":20.844999999999999,"15min":25.013000000000002},"min":{"1min":7.3049999999999997,"5min":7.3049999999999997,"15min":7.3049999999999997},"max":{"1min":91.646000000000001,"5min":177.596,"15min":177.596},"last":14.180999999999999},{"interface":"front","average":{"1min":21.826000000000001,"5min":21.407,"15min":26.369},"min":{"1min":7.476,"5min":7.476,"15min":7.476},"max":{"1min":101.158,"5min":231.03800000000001,"15min":231.03800000000001},"last":6.891}]}]},{"osd":4,"up_from":20,"seq":85899346001,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":610304,"data_stored":266607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":8.0920000000000005,"5min":8.8539999999999992,"15min":11.156000000000001},"min":{"1min":5.0110000000000001,"5min":4.5439999999999996,"15min":4.5439999999999996},"max":{"1min":11.35,"5min":35.052,"15min":35.817999999999998},"last":5.0060000000000002},{"interface":"front","average":{"1min":7.8849999999999998,"5min":9.2420000000000009,"15min":11.247},"min":{"1min":4.6500000000000004,"5min":4.6500000000000004,"15min":4.6500000000000004},"max":{"1min":16.132999999999999,"5min":35.091999999999999,"15min":35.091999999999999},"last":7.7000000000000002}]},{"osd":1,"last update":"Fri Aug 30 16:04:28 2024","interfaces":[{"interface":"back","average":{"1min":12.51,"5min":12.608000000000001,"15min":11.776},"min":{"1min":5.6879999999999997,"5min":4.7000000000000002,"15min":4.7000000000000002},"max":{"1min":27.940000000000001,"5min":34.347999999999999,"15min":34.347999999999999},"last":20.111000000000001},{"interface":"front","average":{"1min":10.170999999999999,"5min":12.247,"15min":12.433999999999999},"min":{"1min":7.3769999999999998,"5min":6.5590000000000002,"15min":6.5590000000000002},"max":{"1min":14.595000000000001,"5min":35.481000000000002,"15min":35.481000000000002},"last":8.9090000000000007}]},{"osd":2,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":9.6660000000000004,"5min":9.5470000000000006,"15min":9.9280000000000008},"min":{"1min":5.4930000000000003,"5min":5.4279999999999999,"15min":5.4279999999999999},"max":{"1min":30.666,"5min":30.666,"15min":30.666},"last":16.872},{"interface":"front","average":{"1min":11.707000000000001,"5min":12.987,"15min":14.148999999999999},"min":{"1min":7.0629999999999997,"5min":7.0629999999999997,"15min":7.0629999999999997},"max":{"1min":21.355,"5min":29.079999999999998,"15min":30.629000000000001},"last":15.125}]},{"osd":3,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":11.263,"5min":11.942,"15min":12.234999999999999},"min":{"1min":7.476,"5min":5.8010000000000002,"15min":5.8010000000000002},"max":{"1min":28.315999999999999,"5min":29.715,"15min":29.715},"last":7.1260000000000003},{"interface":"front","average":{"1min":13.385,"5min":14.148999999999999,"15min":15.195},"min":{"1min":7.8220000000000001,"5min":7.5830000000000002,"15min":7.5830000000000002},"max":{"1min":27.321999999999999,"5min":27.321999999999999,"15min":31.73},"last":15.651}]},{"osd":5,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":14.595000000000001,"5min":13.505000000000001,"15min":11.943},"min":{"1min":9.1199999999999992,"5min":6.8639999999999999,"15min":6.5010000000000003},"max":{"1min":31.565000000000001,"5min":42.143000000000001,"15min":42.143000000000001},"last":18.326000000000001},{"interface":"front","average":{"1min":12.859,"5min":13.661,"15min":14.194000000000001},"min":{"1min":6.5579999999999998,"5min":6.5579999999999998,"15min":6.5579999999999998},"max":{"1min":29.210999999999999,"5min":43.917000000000002,"15min":43.917000000000002},"last":21.623999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:23 2024","interfaces":[{"interface":"back","average":{"1min":13.481,"5min":14.797000000000001,"15min":15.696999999999999},"min":{"1min":8.0359999999999996,"5min":6.5289999999999999,"15min":6.5289999999999999},"max":{"1min":20.704000000000001,"5min":37.215000000000003,"15min":37.215000000000003},"last":17.648},{"interface":"front","average":{"1min":16.030999999999999,"5min":16.199000000000002,"15min":17.021999999999998},"min":{"1min":9.8300000000000001,"5min":8.7870000000000008,"15min":8.7870000000000008},"max":{"1min":24.986999999999998,"5min":37.753999999999998,"15min":37.753999999999998},"last":23.215}]},{"osd":7,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":14.657,"15min":15.507999999999999},"min":{"1min":7.9950000000000001,"5min":7.7130000000000001,"15min":7.7130000000000001},"max":{"1min":21.992999999999999,"5min":33.314,"15min":34.551000000000002},"last":27.103000000000002},{"interface":"front","average":{"1min":12.443,"5min":15.465,"15min":16.873999999999999},"min":{"1min":6.375,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":18.116,"5min":35.701999999999998,"15min":35.701999999999998},"last":21.116}]}]},{"osd":2,"up_from":0,"seq":68719476815,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":15,"seq":64424509518,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":266607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:04:33 2024","interfaces":[{"interface":"back","average":{"1min":10.163,"5min":10.477,"15min":9.9339999999999993},"min":{"1min":4.7489999999999997,"5min":4.7489999999999997,"15min":4.7489999999999997},"max":{"1min":40.735999999999997,"5min":40.735999999999997,"15min":40.735999999999997},"last":5.8940000000000001},{"interface":"front","average":{"1min":12.625999999999999,"5min":11.563000000000001,"15min":11.423},"min":{"1min":4.383,"5min":4.383,"15min":4.383},"max":{"1min":42.387,"5min":42.387,"15min":42.387},"last":5.0069999999999997}]},{"osd":2,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":10.721,"5min":11.506,"15min":11.308},"min":{"1min":6.3899999999999997,"5min":5.3689999999999998,"15min":3.3370000000000002},"max":{"1min":24.170999999999999,"5min":46.768999999999998,"15min":46.768999999999998},"last":12.143000000000001},{"interface":"front","average":{"1min":13.676,"5min":13.199999999999999,"15min":14.022},"min":{"1min":6.8860000000000001,"5min":6.2249999999999996,"15min":5.702},"max":{"1min":31.361000000000001,"5min":43.921999999999997,"15min":43.921999999999997},"last":9.6780000000000008}]},{"osd":3,"last update":"Fri Aug 30 16:04:34 2024","interfaces":[{"interface":"back","average":{"1min":13.814,"5min":16.356000000000002,"15min":22.327999999999999},"min":{"1min":6.6680000000000001,"5min":6.2839999999999998,"15min":6.2839999999999998},"max":{"1min":38.140999999999998,"5min":200.155,"15min":200.155},"last":10.26},{"interface":"front","average":{"1min":15.974,"5min":20.137,"15min":27.510999999999999},"min":{"1min":8.4339999999999993,"5min":7.0940000000000003,"15min":7.0940000000000003},"max":{"1min":35.576999999999998,"5min":267.03100000000001,"15min":267.03100000000001},"last":23.777999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:04:34 2024","interfaces":[{"interface":"back","average":{"1min":11.683,"5min":14.086,"15min":15.364000000000001},"min":{"1min":6.0970000000000004,"5min":6.0970000000000004,"15min":6.0970000000000004},"max":{"1min":23.608000000000001,"5min":56.200000000000003,"15min":56.200000000000003},"last":8.625},{"interface":"front","average":{"1min":15.760999999999999,"5min":16.109000000000002,"15min":15.863},"min":{"1min":9.298,"5min":8.609,"15min":8.609},"max":{"1min":24.710000000000001,"5min":48.534999999999997,"15min":48.534999999999997},"last":11.393000000000001}]},{"osd":5,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":14.43,"5min":15.682,"15min":16.082999999999998},"min":{"1min":9.8059999999999992,"5min":5.9199999999999999,"15min":5.6920000000000002},"max":{"1min":40.006,"5min":43.308,"15min":43.308},"last":24.443999999999999},{"interface":"front","average":{"1min":16.951000000000001,"5min":17.693000000000001,"15min":17.853999999999999},"min":{"1min":9.9100000000000001,"5min":9.2149999999999999,"15min":4.75},"max":{"1min":49.575000000000003,"5min":49.575000000000003,"15min":49.575000000000003},"last":12.695}]},{"osd":6,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":13.808999999999999,"5min":14.292,"15min":14.385999999999999},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":39.460999999999999,"5min":59.328000000000003,"15min":59.328000000000003},"last":23.652999999999999},{"interface":"front","average":{"1min":19.225000000000001,"5min":18.331,"15min":18.533999999999999},"min":{"1min":10.781000000000001,"5min":10.005000000000001,"15min":10.005000000000001},"max":{"1min":46.237000000000002,"5min":58.567,"15min":58.567},"last":38.289000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:04:33 2024","interfaces":[{"interface":"back","average":{"1min":18.983000000000001,"5min":19.995000000000001,"15min":21.375},"min":{"1min":12.195,"5min":9.1180000000000003,"15min":9.1180000000000003},"max":{"1min":34.927999999999997,"5min":59.515999999999998,"15min":59.515999999999998},"last":13.212999999999999},{"interface":"front","average":{"1min":12.894,"5min":14.285,"15min":15.276999999999999},"min":{"1min":7.6379999999999999,"5min":7.1619999999999999,"15min":7.1619999999999999},"max":{"1min":24.966000000000001,"5min":58.807000000000002,"15min":58.807000000000002},"last":8.3520000000000003}]}]},{"osd":5,"up_from":0,"seq":55834574926,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":397840,"data_compressed":689,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:05:29.946 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-08-30T16:05:29.946 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T16:05:29.946 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-08-30T16:05:29.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 2 2024-08-30T16:05:30.507 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T16:05:30.504+0000 1008f640 -1 mon.a@0(leader).osd e109 definitely_dead 0 2024-08-30T16:05:30.960 INFO:teuthology.orchestra.run.smithi050.stderr:marked in osd.2. 2024-08-30T16:05:30.972 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-08-30T16:05:33.515 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:56.121 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:33.782 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:56.170 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:33.782 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:56.184 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:33.782 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:55.148 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:33.783 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:55.150 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:33.783 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:01:55.163 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:33.783 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:54.776 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:33.783 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:54.777 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:33.783 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:54.790 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:35.973 INFO:tasks.thrashosds.thrasher:inning osd 2024-08-30T16:05:35.973 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-08-30T16:05:35.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-08-30T16:05:36.578 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T16:05:36.576+0000 1008f640 -1 mon.a@0(leader).osd e111 definitely_dead 0 2024-08-30T16:05:37.083 INFO:teuthology.orchestra.run.smithi050.stderr:marked in osd.5. 2024-08-30T16:05:37.099 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-08-30T16:05:37.099 INFO:tasks.thrashosds.thrasher:inning osd 2024-08-30T16:05:37.099 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-08-30T16:05:37.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 6 2024-08-30T16:05:37.703 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T16:05:37.700+0000 1008f640 -1 mon.a@0(leader).osd e112 definitely_dead 0 2024-08-30T16:05:38.122 INFO:teuthology.orchestra.run.smithi050.stderr:marked in osd.6. 2024-08-30T16:05:38.137 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-08-30T16:05:38.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-08-30T16:05:38.503 INFO:teuthology.orchestra.run.smithi050.stdout:mon.a: {} 2024-08-30T16:05:38.503 INFO:teuthology.orchestra.run.smithi050.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-30T16:05:38.643 INFO:teuthology.orchestra.run.smithi050.stdout:mon.b: {} 2024-08-30T16:05:38.644 INFO:teuthology.orchestra.run.smithi050.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-30T16:05:38.827 INFO:teuthology.orchestra.run.smithi050.stdout:mon.c: {} 2024-08-30T16:05:38.827 INFO:teuthology.orchestra.run.smithi050.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-30T16:05:38.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-08-30T16:05:40.539 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.0 to 1 (10000) 2024-08-30T16:05:40.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-08-30T16:05:41.621 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-08-30T16:05:41.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-08-30T16:05:42.595 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.1 to 1 (10000) 2024-08-30T16:05:42.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-08-30T16:05:43.692 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-08-30T16:05:43.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-08-30T16:05:44.828 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.2 to 1 (10000) 2024-08-30T16:05:44.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-08-30T16:05:45.981 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-08-30T16:05:45.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-08-30T16:05:46.998 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.3 to 1 (10000) 2024-08-30T16:05:47.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-08-30T16:05:48.254 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-08-30T16:05:48.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-08-30T16:05:48.779 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:10.316 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:48.780 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:10.317 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:48.780 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:11.385 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:48.783 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:11.388 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:48.803 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:10.340 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:48.805 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:11.411 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:48.839 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:09.998 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:48.841 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:10.001 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:48.865 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:10.025 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:49.824 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.4 to 1 (10000) 2024-08-30T16:05:49.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-08-30T16:05:50.829 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-08-30T16:05:50.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-08-30T16:05:51.917 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.5 to 1 (10000) 2024-08-30T16:05:51.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-08-30T16:05:53.012 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-08-30T16:05:53.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-08-30T16:05:53.613 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:16.218 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:53.615 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:16.220 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:53.629 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:16.234 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:53.665 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:14.824 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:53.667 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:14.827 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:53.668 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:15.205 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:53.670 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:15.207 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:53.680 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:14.839 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:53.695 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:15.232 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:05:54.639 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.7 to 1 (10000) 2024-08-30T16:05:54.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-08-30T16:05:56.233 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-08-30T16:05:56.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-08-30T16:05:57.257 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.6 to 1 (10000) 2024-08-30T16:05:57.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-08-30T16:05:58.510 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-08-30T16:05:58.523 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-08-30T16:05:58.523 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-08-30T16:05:58.523 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-08-30T16:05:58.523 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-08-30T16:05:58.523 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-08-30T16:05:58.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-30T16:05:59.098 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:05:59.098 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":130,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T16:05:58.319477+0000","last_up_change":"2024-08-30T16:05:24.091073+0000","last_in_change":"2024-08-30T16:05:38.025676+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-30T15:59:08.355540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-30T15:59:21.696796+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":2.630000114440918,"raw_score_stable":2.630000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6812","nonce":1957399494}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6813","nonce":1957399494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6815","nonce":1957399494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6814","nonce":1957399494}]},"public_addr":"172.21.15.50:6812/1957399494","cluster_addr":"172.21.15.50:6813/1957399494","heartbeat_back_addr":"172.21.15.50:6815/1957399494","heartbeat_front_addr":"172.21.15.50:6814/1957399494","state":["exists","up"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6808","nonce":3208178756}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6809","nonce":3208178756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6811","nonce":3208178756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6810","nonce":3208178756}]},"public_addr":"172.21.15.50:6808/3208178756","cluster_addr":"172.21.15.50:6809/3208178756","heartbeat_back_addr":"172.21.15.50:6811/3208178756","heartbeat_front_addr":"172.21.15.50:6810/3208178756","state":["exists","up"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6804","nonce":287018800}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6805","nonce":287018800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6807","nonce":287018800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6806","nonce":287018800}]},"public_addr":"172.21.15.50:6804/287018800","cluster_addr":"172.21.15.50:6805/287018800","heartbeat_back_addr":"172.21.15.50:6807/287018800","heartbeat_front_addr":"172.21.15.50:6806/287018800","state":["exists","up"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6800","nonce":1140717714}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6801","nonce":1140717714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6803","nonce":1140717714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6802","nonce":1140717714}]},"public_addr":"172.21.15.50:6800/1140717714","cluster_addr":"172.21.15.50:6801/1140717714","heartbeat_back_addr":"172.21.15.50:6803/1140717714","heartbeat_front_addr":"172.21.15.50:6802/1140717714","state":["exists","up"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":1468073345}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":1468073345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":1468073345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":1468073345}]},"public_addr":"172.21.15.94:6805/1468073345","cluster_addr":"172.21.15.94:6806/1468073345","heartbeat_back_addr":"172.21.15.94:6808/1468073345","heartbeat_front_addr":"172.21.15.94:6807/1468073345","state":["exists","up"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":3876025616}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":3876025616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":3876025616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":3876025616}]},"public_addr":"172.21.15.94:6809/3876025616","cluster_addr":"172.21.15.94:6810/3876025616","heartbeat_back_addr":"172.21.15.94:6812/3876025616","heartbeat_front_addr":"172.21.15.94:6811/3876025616","state":["exists","up"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":74,"up_from":105,"up_thru":0,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":1984944988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":1984944988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":1984944988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":1984944988}]},"public_addr":"172.21.15.94:6801/1984944988","cluster_addr":"172.21.15.94:6802/1984944988","heartbeat_back_addr":"172.21.15.94:6804/1984944988","heartbeat_front_addr":"172.21.15.94:6803/1984944988","state":["exists","up"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":68721277}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":68721277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6816","nonce":68721277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":68721277}]},"public_addr":"172.21.15.94:6813/68721277","cluster_addr":"172.21.15.94:6814/68721277","heartbeat_back_addr":"172.21.15.94:6816/68721277","heartbeat_front_addr":"172.21.15.94:6815/68721277","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.911895+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.786543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.197952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.680847+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:01.937600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:02.082761+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.401025+0000","dead_epoch":74},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.129371+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.3","osds":[7,0]},{"pgid":"2.5","osds":[2,5]},{"pgid":"2.7","osds":[7,0]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T16:05:59.110 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-08-30T16:05:59.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-30T16:05:59.692 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:05:59.692 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":130,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T16:05:58.319477+0000","last_up_change":"2024-08-30T16:05:24.091073+0000","last_in_change":"2024-08-30T16:05:38.025676+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-30T15:59:08.355540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-30T15:59:21.696796+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":2.630000114440918,"raw_score_stable":2.630000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6812","nonce":1957399494}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6813","nonce":1957399494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6815","nonce":1957399494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6814","nonce":1957399494}]},"public_addr":"172.21.15.50:6812/1957399494","cluster_addr":"172.21.15.50:6813/1957399494","heartbeat_back_addr":"172.21.15.50:6815/1957399494","heartbeat_front_addr":"172.21.15.50:6814/1957399494","state":["exists","up"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6808","nonce":3208178756}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6809","nonce":3208178756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6811","nonce":3208178756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6810","nonce":3208178756}]},"public_addr":"172.21.15.50:6808/3208178756","cluster_addr":"172.21.15.50:6809/3208178756","heartbeat_back_addr":"172.21.15.50:6811/3208178756","heartbeat_front_addr":"172.21.15.50:6810/3208178756","state":["exists","up"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6804","nonce":287018800}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6805","nonce":287018800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6807","nonce":287018800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6806","nonce":287018800}]},"public_addr":"172.21.15.50:6804/287018800","cluster_addr":"172.21.15.50:6805/287018800","heartbeat_back_addr":"172.21.15.50:6807/287018800","heartbeat_front_addr":"172.21.15.50:6806/287018800","state":["exists","up"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6800","nonce":1140717714}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6801","nonce":1140717714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6803","nonce":1140717714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6802","nonce":1140717714}]},"public_addr":"172.21.15.50:6800/1140717714","cluster_addr":"172.21.15.50:6801/1140717714","heartbeat_back_addr":"172.21.15.50:6803/1140717714","heartbeat_front_addr":"172.21.15.50:6802/1140717714","state":["exists","up"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":1468073345}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":1468073345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":1468073345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":1468073345}]},"public_addr":"172.21.15.94:6805/1468073345","cluster_addr":"172.21.15.94:6806/1468073345","heartbeat_back_addr":"172.21.15.94:6808/1468073345","heartbeat_front_addr":"172.21.15.94:6807/1468073345","state":["exists","up"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":3876025616}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":3876025616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":3876025616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":3876025616}]},"public_addr":"172.21.15.94:6809/3876025616","cluster_addr":"172.21.15.94:6810/3876025616","heartbeat_back_addr":"172.21.15.94:6812/3876025616","heartbeat_front_addr":"172.21.15.94:6811/3876025616","state":["exists","up"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":74,"up_from":105,"up_thru":0,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":1984944988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":1984944988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":1984944988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":1984944988}]},"public_addr":"172.21.15.94:6801/1984944988","cluster_addr":"172.21.15.94:6802/1984944988","heartbeat_back_addr":"172.21.15.94:6804/1984944988","heartbeat_front_addr":"172.21.15.94:6803/1984944988","state":["exists","up"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":68721277}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":68721277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6816","nonce":68721277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":68721277}]},"public_addr":"172.21.15.94:6813/68721277","cluster_addr":"172.21.15.94:6814/68721277","heartbeat_back_addr":"172.21.15.94:6816/68721277","heartbeat_front_addr":"172.21.15.94:6815/68721277","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.911895+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.786543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.197952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.680847+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:01.937600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:02.082761+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.401025+0000","dead_epoch":74},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.129371+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.3","osds":[7,0]},{"pgid":"2.5","osds":[2,5]},{"pgid":"2.7","osds":[7,0]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T16:05:59.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-08-30T16:05:59.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-08-30T16:05:59.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-08-30T16:05:59.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-08-30T16:05:59.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-08-30T16:05:59.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-08-30T16:05:59.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-08-30T16:05:59.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-08-30T16:06:00.069 INFO:teuthology.orchestra.run.smithi050.stdout:90194313305 2024-08-30T16:06:00.070 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-30T16:06:00.086 INFO:teuthology.orchestra.run.smithi050.stdout:55834574934 2024-08-30T16:06:00.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-30T16:06:00.101 INFO:teuthology.orchestra.run.smithi050.stdout:90194313305 2024-08-30T16:06:00.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-08-30T16:06:00.165 INFO:teuthology.orchestra.run.smithi050.stdout:94489280601 2024-08-30T16:06:00.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-30T16:06:00.184 INFO:teuthology.orchestra.run.smithi050.stdout:85899346008 2024-08-30T16:06:00.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-30T16:06:00.186 INFO:teuthology.orchestra.run.smithi050.stdout:450971566090 2024-08-30T16:06:00.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-08-30T16:06:00.216 INFO:teuthology.orchestra.run.smithi050.stdout:68719476822 2024-08-30T16:06:00.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-30T16:06:00.217 INFO:teuthology.orchestra.run.smithi050.stdout:64424509525 2024-08-30T16:06:00.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-30T16:06:00.840 INFO:teuthology.orchestra.run.smithi050.stdout:90194313303 2024-08-30T16:06:00.857 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313305 got 90194313303 for osd.3 2024-08-30T16:06:01.042 INFO:teuthology.orchestra.run.smithi050.stdout:85899346007 2024-08-30T16:06:01.055 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899346008 got 85899346007 for osd.4 2024-08-30T16:06:01.268 INFO:teuthology.orchestra.run.smithi050.stdout:55834574932 2024-08-30T16:06:01.285 INFO:teuthology.orchestra.run.smithi050.stdout:90194313303 2024-08-30T16:06:01.286 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574934 got 55834574932 for osd.5 2024-08-30T16:06:01.303 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313305 got 90194313303 for osd.7 2024-08-30T16:06:01.477 INFO:teuthology.orchestra.run.smithi050.stdout:68719476821 2024-08-30T16:06:01.490 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476822 got 68719476821 for osd.2 2024-08-30T16:06:01.517 INFO:teuthology.orchestra.run.smithi050.stdout:450971566089 2024-08-30T16:06:01.531 INFO:tasks.ceph.ceph_manager.ceph:need seq 450971566090 got 450971566089 for osd.6 2024-08-30T16:06:01.570 INFO:teuthology.orchestra.run.smithi050.stdout:64424509524 2024-08-30T16:06:01.583 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509525 got 64424509524 for osd.0 2024-08-30T16:06:01.588 INFO:teuthology.orchestra.run.smithi050.stdout:94489280600 2024-08-30T16:06:01.603 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280601 got 94489280600 for osd.1 2024-08-30T16:06:01.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-30T16:06:02.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-30T16:06:02.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-30T16:06:02.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-08-30T16:06:02.415 INFO:teuthology.orchestra.run.smithi050.stdout:90194313304 2024-08-30T16:06:02.428 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313305 got 90194313304 for osd.3 2024-08-30T16:06:02.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-30T16:06:02.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-08-30T16:06:02.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-30T16:06:02.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-30T16:06:02.690 INFO:teuthology.orchestra.run.smithi050.stdout:85899346007 2024-08-30T16:06:02.711 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899346008 got 85899346007 for osd.4 2024-08-30T16:06:02.865 INFO:teuthology.orchestra.run.smithi050.stdout:55834574933 2024-08-30T16:06:02.881 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574934 got 55834574933 for osd.5 2024-08-30T16:06:03.255 INFO:teuthology.orchestra.run.smithi050.stdout:90194313305 2024-08-30T16:06:03.269 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313305 got 90194313305 for osd.7 2024-08-30T16:06:03.269 DEBUG:teuthology.parallel:result is None 2024-08-30T16:06:03.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-30T16:06:03.552 INFO:teuthology.orchestra.run.smithi050.stdout:450971566090 2024-08-30T16:06:03.568 INFO:tasks.ceph.ceph_manager.ceph:need seq 450971566090 got 450971566090 for osd.6 2024-08-30T16:06:03.569 DEBUG:teuthology.parallel:result is None 2024-08-30T16:06:03.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-30T16:06:03.737 INFO:teuthology.orchestra.run.smithi050.stdout:68719476822 2024-08-30T16:06:03.749 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476822 got 68719476822 for osd.2 2024-08-30T16:06:03.749 DEBUG:teuthology.parallel:result is None 2024-08-30T16:06:03.797 INFO:teuthology.orchestra.run.smithi050.stdout:94489280602 2024-08-30T16:06:03.814 INFO:teuthology.orchestra.run.smithi050.stdout:64424509526 2024-08-30T16:06:03.817 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280601 got 94489280602 for osd.1 2024-08-30T16:06:03.817 DEBUG:teuthology.parallel:result is None 2024-08-30T16:06:03.828 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509525 got 64424509526 for osd.0 2024-08-30T16:06:03.828 DEBUG:teuthology.parallel:result is None 2024-08-30T16:06:03.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-30T16:06:04.009 INFO:teuthology.orchestra.run.smithi050.stdout:90194313305 2024-08-30T16:06:04.022 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313305 got 90194313305 for osd.3 2024-08-30T16:06:04.023 DEBUG:teuthology.parallel:result is None 2024-08-30T16:06:04.414 INFO:teuthology.orchestra.run.smithi050.stdout:85899346008 2024-08-30T16:06:04.429 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899346008 got 85899346008 for osd.4 2024-08-30T16:06:04.429 DEBUG:teuthology.parallel:result is None 2024-08-30T16:06:04.481 INFO:teuthology.orchestra.run.smithi050.stdout:55834574934 2024-08-30T16:06:04.494 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574934 got 55834574934 for osd.5 2024-08-30T16:06:04.494 DEBUG:teuthology.parallel:result is None 2024-08-30T16:06:04.494 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-08-30T16:06:04.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:06:05.035 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:06:05.035 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:06:05.048 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-08-30T16:06:03.938249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1591398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222156,"kb_used_data":5480,"kb_used_omap":15,"kb_used_meta":216560,"kb_avail":754752564,"statfs":{"total":773094113280,"available":772866625536,"internally_reserved":0,"allocated":5611520,"data_stored":2880942,"data_compressed":1558,"data_compressed_allocated":49152,"data_compressed_original":786432,"omap_allocated":15591,"internal_metadata":221758233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.751840"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.467046+0000","last_change":"2024-08-30T16:04:11.379068+0000","last_active":"2024-08-30T16:05:57.467046+0000","last_peered":"2024-08-30T16:05:57.467046+0000","last_clean":"2024-08-30T16:05:57.467046+0000","last_became_active":"2024-08-30T16:04:11.370524+0000","last_became_peered":"2024-08-30T16:04:11.370524+0000","last_unstale":"2024-08-30T16:05:57.467046+0000","last_undegraded":"2024-08-30T16:05:57.467046+0000","last_fullsized":"2024-08-30T16:05:57.467046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:58:06.441243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.022669+0000","last_change":"2024-08-30T16:04:38.088583+0000","last_active":"2024-08-30T16:05:58.022669+0000","last_peered":"2024-08-30T16:05:58.022669+0000","last_clean":"2024-08-30T16:05:58.022669+0000","last_became_active":"2024-08-30T16:04:38.082821+0000","last_became_peered":"2024-08-30T16:04:38.082821+0000","last_unstale":"2024-08-30T16:05:58.022669+0000","last_undegraded":"2024-08-30T16:05:58.022669+0000","last_fullsized":"2024-08-30T16:05:58.022669+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:40:53.077427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.520451+0000","last_change":"2024-08-30T16:05:38.369602+0000","last_active":"2024-08-30T16:05:57.520451+0000","last_peered":"2024-08-30T16:05:57.520451+0000","last_clean":"2024-08-30T16:05:57.520451+0000","last_became_active":"2024-08-30T16:05:38.361240+0000","last_became_peered":"2024-08-30T16:05:38.361240+0000","last_unstale":"2024-08-30T16:05:57.520451+0000","last_undegraded":"2024-08-30T16:05:57.520451+0000","last_fullsized":"2024-08-30T16:05:57.520451+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:02:07.953195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.259961+0000","last_change":"2024-08-30T16:05:39.779631+0000","last_active":"2024-08-30T16:05:57.259961+0000","last_peered":"2024-08-30T16:05:57.259961+0000","last_clean":"2024-08-30T16:05:57.259961+0000","last_became_active":"2024-08-30T16:05:39.773479+0000","last_became_peered":"2024-08-30T16:05:39.773479+0000","last_unstale":"2024-08-30T16:05:57.259961+0000","last_undegraded":"2024-08-30T16:05:57.259961+0000","last_fullsized":"2024-08-30T16:05:57.259961+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:29:29.455454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.255363+0000","last_change":"2024-08-30T16:05:40.031358+0000","last_active":"2024-08-30T16:05:57.255363+0000","last_peered":"2024-08-30T16:05:57.255363+0000","last_clean":"2024-08-30T16:05:57.255363+0000","last_became_active":"2024-08-30T16:05:39.776453+0000","last_became_peered":"2024-08-30T16:05:39.776453+0000","last_unstale":"2024-08-30T16:05:57.255363+0000","last_undegraded":"2024-08-30T16:05:57.255363+0000","last_fullsized":"2024-08-30T16:05:57.255363+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:06:14.092608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.462397+0000","last_change":"2024-08-30T16:05:38.365381+0000","last_active":"2024-08-30T16:05:57.462397+0000","last_peered":"2024-08-30T16:05:57.462397+0000","last_clean":"2024-08-30T16:05:57.462397+0000","last_became_active":"2024-08-30T16:05:38.358584+0000","last_became_peered":"2024-08-30T16:05:38.358584+0000","last_unstale":"2024-08-30T16:05:57.462397+0000","last_undegraded":"2024-08-30T16:05:57.462397+0000","last_fullsized":"2024-08-30T16:05:57.462397+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:28:34.896593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":141,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.257472+0000","last_change":"2024-08-30T16:05:38.364964+0000","last_active":"2024-08-30T16:05:57.257472+0000","last_peered":"2024-08-30T16:05:57.257472+0000","last_clean":"2024-08-30T16:05:57.257472+0000","last_became_active":"2024-08-30T16:05:38.358598+0000","last_became_peered":"2024-08-30T16:05:38.358598+0000","last_unstale":"2024-08-30T16:05:57.257472+0000","last_undegraded":"2024-08-30T16:05:57.257472+0000","last_fullsized":"2024-08-30T16:05:57.257472+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T03:36:43.310274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":61,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.904323+0000","last_change":"2024-08-30T16:04:08.039936+0000","last_active":"2024-08-30T16:05:58.904323+0000","last_peered":"2024-08-30T16:05:58.904323+0000","last_clean":"2024-08-30T16:05:58.904323+0000","last_became_active":"2024-08-30T16:04:07.760364+0000","last_became_peered":"2024-08-30T16:04:07.760364+0000","last_unstale":"2024-08-30T16:05:58.904323+0000","last_undegraded":"2024-08-30T16:05:58.904323+0000","last_fullsized":"2024-08-30T16:05:58.904323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:13:37.016725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":160,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.024190+0000","last_change":"2024-08-30T16:05:38.553500+0000","last_active":"2024-08-30T16:05:58.024190+0000","last_peered":"2024-08-30T16:05:58.024190+0000","last_clean":"2024-08-30T16:05:58.024190+0000","last_became_active":"2024-08-30T16:05:38.360837+0000","last_became_peered":"2024-08-30T16:05:38.360837+0000","last_unstale":"2024-08-30T16:05:58.024190+0000","last_undegraded":"2024-08-30T16:05:58.024190+0000","last_fullsized":"2024-08-30T16:05:58.024190+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:15:13.872133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1591360,"data_compressed":3116,"data_compressed_allocated":98304,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280602,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27732,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344108,"statfs":{"total":96636764160,"available":96608366592,"internally_reserved":0,"allocated":724992,"data_stored":658493,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.896000000000001,"5min":11.542,"15min":13.461},"min":{"1min":5.5519999999999996,"5min":4.8049999999999997,"15min":4.8049999999999997},"max":{"1min":25.129999999999999,"5min":51.027000000000001,"15min":57.177999999999997},"last":6.0860000000000003},{"interface":"front","average":{"1min":9.5020000000000007,"5min":11.342000000000001,"15min":12.673999999999999},"min":{"1min":4.5759999999999996,"5min":4.5759999999999996,"15min":4.5759999999999996},"max":{"1min":16.521000000000001,"5min":54.682000000000002,"15min":58.024999999999999},"last":8.3930000000000007}]},{"osd":2,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.157,"5min":10.984999999999999,"15min":10.614000000000001},"min":{"1min":6.0880000000000001,"5min":5.5780000000000003,"15min":5.5780000000000003},"max":{"1min":15.81,"5min":73.399000000000001,"15min":73.399000000000001},"last":8.0540000000000003},{"interface":"front","average":{"1min":13.478,"5min":14.470000000000001,"15min":14.875},"min":{"1min":6.5640000000000001,"5min":6.0049999999999999,"15min":6.0049999999999999},"max":{"1min":30.704999999999998,"5min":75.120000000000005,"15min":75.120000000000005},"last":10.186999999999999}]},{"osd":3,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":13.753,"5min":15.361000000000001,"15min":16.896000000000001},"min":{"1min":9.1199999999999992,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":29.506,"5min":66.869,"15min":66.869},"last":10.727},{"interface":"front","average":{"1min":10.523999999999999,"5min":13.089,"15min":14.315},"min":{"1min":5.8959999999999999,"5min":5.8959999999999999,"15min":5.8959999999999999},"max":{"1min":16.196000000000002,"5min":104.38200000000001,"15min":104.38200000000001},"last":9.6280000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":11.037000000000001,"5min":14.484999999999999,"15min":14.68},"min":{"1min":7.5590000000000002,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":18.202999999999999,"5min":66.340999999999994,"15min":66.340999999999994},"last":13.754},{"interface":"front","average":{"1min":9.5540000000000003,"5min":12.513999999999999,"15min":12.881},"min":{"1min":6.774,"5min":6.774,"15min":6.774},"max":{"1min":14.335000000000001,"5min":29.818999999999999,"15min":29.818999999999999},"last":13.16}]},{"osd":5,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":13.954000000000001,"5min":15.705,"15min":16.510999999999999},"min":{"1min":7.21,"5min":7.21,"15min":7.21},"max":{"1min":31.545000000000002,"5min":55.784999999999997,"15min":55.784999999999997},"last":11.273999999999999},{"interface":"front","average":{"1min":11.759,"5min":17.265999999999998,"15min":18.946000000000002},"min":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"max":{"1min":16.829999999999998,"5min":68.004999999999995,"15min":68.004999999999995},"last":16.846}]},{"osd":6,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":15.708,"5min":16.609000000000002,"15min":17.382000000000001},"min":{"1min":10.180999999999999,"5min":9.1929999999999996,"15min":9.1929999999999996},"max":{"1min":27.353999999999999,"5min":58.201999999999998,"15min":58.201999999999998},"last":14.301},{"interface":"front","average":{"1min":16.324999999999999,"5min":16.748000000000001,"15min":17.001000000000001},"min":{"1min":9.6980000000000004,"5min":8.8390000000000004,"15min":8.8390000000000004},"max":{"1min":26.257999999999999,"5min":37.253,"15min":37.253},"last":15.581}]},{"osd":7,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":16.931000000000001,"5min":17.739999999999998,"15min":19.425999999999998},"min":{"1min":10.352,"5min":8.2050000000000001,"15min":8.2050000000000001},"max":{"1min":34.345999999999997,"5min":68.950000000000003,"15min":68.950000000000003},"last":16.158000000000001},{"interface":"front","average":{"1min":13.039999999999999,"5min":16.266999999999999,"15min":18.443999999999999},"min":{"1min":9.4529999999999994,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":17.625,"5min":36.186,"15min":59.079000000000001},"last":17.533999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313306,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":7.9219999999999997,"5min":8.6739999999999995,"15min":8.8670000000000009},"min":{"1min":4.3789999999999996,"5min":4.3789999999999996,"15min":4.3789999999999996},"max":{"1min":20.73,"5min":25.759,"15min":30.600999999999999},"last":14.135999999999999},{"interface":"front","average":{"1min":7.1589999999999998,"5min":8.8819999999999997,"15min":9.0419999999999998},"min":{"1min":4.8460000000000001,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":12.57,"5min":28.077000000000002,"15min":28.077000000000002},"last":6.0970000000000004}]},{"osd":1,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":11.804,"5min":12.818,"15min":11.496},"min":{"1min":7.0590000000000002,"5min":5.7009999999999996,"15min":5.7009999999999996},"max":{"1min":18.989999999999998,"5min":83.581000000000003,"15min":83.581000000000003},"last":14.663},{"interface":"front","average":{"1min":9.2690000000000001,"5min":11.781000000000001,"15min":12.198},"min":{"1min":5.6230000000000002,"5min":5.6230000000000002,"15min":5.6230000000000002},"max":{"1min":19.524999999999999,"5min":81.722999999999999,"15min":81.722999999999999},"last":7.5270000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":14.83,"5min":13.890000000000001,"15min":13.689},"min":{"1min":8.5039999999999996,"5min":5.4850000000000003,"15min":5.4850000000000003},"max":{"1min":25.885999999999999,"5min":42.506,"15min":42.506},"last":18.111999999999998},{"interface":"front","average":{"1min":13.273999999999999,"5min":13.69,"15min":16.242000000000001},"min":{"1min":6.8029999999999999,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":24.309000000000001,"5min":43.484999999999999,"15min":43.484999999999999},"last":15.276}]},{"osd":3,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":15.515000000000001,"5min":15.986000000000001,"15min":16.391999999999999},"min":{"1min":7.9299999999999997,"5min":7.9299999999999997,"15min":7.9299999999999997},"max":{"1min":54.076999999999998,"5min":65.739999999999995,"15min":65.739999999999995},"last":8.4290000000000003},{"interface":"front","average":{"1min":12.406000000000001,"5min":13.285,"15min":12.871},"min":{"1min":5.6070000000000002,"5min":5.6070000000000002,"15min":5.6070000000000002},"max":{"1min":55.978000000000002,"5min":64.980000000000004,"15min":64.980000000000004},"last":17.135000000000002}]},{"osd":4,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":13.824,"15min":12.827},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.4880000000000004},"max":{"1min":23.213999999999999,"5min":41.631,"15min":41.631},"last":9.6419999999999995},{"interface":"front","average":{"1min":14.590999999999999,"5min":15.433,"15min":17.003},"min":{"1min":9.2789999999999999,"5min":8.8729999999999993,"15min":8.8729999999999993},"max":{"1min":25.149999999999999,"5min":32.526000000000003,"15min":37.393000000000001},"last":12.411}]},{"osd":5,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.68,"5min":12.377000000000001,"15min":11.679},"min":{"1min":6.8259999999999996,"5min":6.2309999999999999,"15min":6.2309999999999999},"max":{"1min":37.372999999999998,"5min":37.372999999999998,"15min":37.372999999999998},"last":16.436},{"interface":"front","average":{"1min":16.558,"5min":15.769,"15min":16.600000000000001},"min":{"1min":9.0429999999999993,"5min":8.2870000000000008,"15min":8.2870000000000008},"max":{"1min":38.484000000000002,"5min":38.618000000000002,"15min":38.618000000000002},"last":10.513}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.326000000000001,"5min":13.771000000000001,"15min":13.208},"min":{"1min":8.0060000000000002,"5min":6.3230000000000004,"15min":6.3230000000000004},"max":{"1min":19.231000000000002,"5min":42.829999999999998,"15min":42.829999999999998},"last":11.218},{"interface":"front","average":{"1min":15.842000000000001,"5min":17.606000000000002,"15min":18.088999999999999},"min":{"1min":11.712,"5min":10.749000000000001,"15min":10.749000000000001},"max":{"1min":22.695,"5min":44.491999999999997,"15min":44.491999999999997},"last":13.292}]}]},{"osd":6,"up_from":105,"seq":450971566091,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":680,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":696320,"data_stored":260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":28176046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.835000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.928000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9849999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.256}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.364000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":696320,"data_stored":260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":9.4120000000000008,"5min":9.5199999999999996,"15min":10.925000000000001},"min":{"1min":5.0830000000000002,"5min":4.3810000000000002,"15min":4.29},"max":{"1min":18.713000000000001,"5min":46.146999999999998,"15min":50.500999999999998},"last":4.7599999999999998},{"interface":"front","average":{"1min":9.7110000000000003,"5min":10.157,"15min":13.363},"min":{"1min":5.9279999999999999,"5min":5.1360000000000001,"15min":5.1360000000000001},"max":{"1min":20.637,"5min":36.720999999999997,"15min":51.965000000000003},"last":9.9149999999999991}]},{"osd":1,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":12.821999999999999,"5min":13.605,"15min":17.742999999999999},"min":{"1min":7.7549999999999999,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":20.864999999999998,"5min":44.862000000000002,"15min":137.56700000000001},"last":7.7279999999999998},{"interface":"front","average":{"1min":10.217000000000001,"5min":12.255000000000001,"15min":15.741},"min":{"1min":6.7839999999999998,"5min":5.0060000000000002,"15min":5.0060000000000002},"max":{"1min":27.032,"5min":47.530999999999999,"15min":136.321},"last":13.763}]},{"osd":2,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":11.897,"5min":12.744999999999999,"15min":15.519},"min":{"1min":5.9109999999999996,"5min":5.9109999999999996,"15min":5.9109999999999996},"max":{"1min":26.527000000000001,"5min":43.869999999999997,"15min":73.831000000000003},"last":11.119999999999999},{"interface":"front","average":{"1min":11.170999999999999,"5min":11.682,"15min":14.923},"min":{"1min":6.984,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":16.887,"5min":26.074999999999999,"15min":71.831999999999994},"last":20.190000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":14.353,"15min":26.016999999999999},"min":{"1min":7.2350000000000003,"5min":6.9199999999999999,"15min":6.9199999999999999},"max":{"1min":25.645,"5min":43.235999999999997,"15min":342.34500000000003},"last":8.9779999999999998},{"interface":"front","average":{"1min":12.157,"5min":14.849,"15min":19.047000000000001},"min":{"1min":7.1050000000000004,"5min":6.7859999999999996,"15min":6.7859999999999996},"max":{"1min":23.939,"5min":63.633000000000003,"15min":127.857},"last":18.582000000000001}]},{"osd":5,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":10.967000000000001,"5min":14.295,"15min":15.500999999999999},"min":{"1min":7.1200000000000001,"5min":6.5,"15min":6.5},"max":{"1min":16.876000000000001,"5min":58.715000000000003,"15min":58.715000000000003},"last":11.959},{"interface":"front","average":{"1min":14.762,"5min":16.952000000000002,"15min":18.649000000000001},"min":{"1min":9.1549999999999994,"5min":7.5869999999999997,"15min":7.2370000000000001},"max":{"1min":29.803999999999998,"5min":58.212000000000003,"15min":58.212000000000003},"last":15.076000000000001}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.295000000000002,"5min":23.492000000000001,"15min":30.053999999999998},"min":{"1min":11.096,"5min":8.6630000000000003,"15min":8.6630000000000003},"max":{"1min":25.298999999999999,"5min":388.96100000000001,"15min":388.96100000000001},"last":13.083},{"interface":"front","average":{"1min":13.961,"5min":24.18,"15min":32.835999999999999},"min":{"1min":9.4079999999999995,"5min":8.5519999999999996,"15min":8.5519999999999996},"max":{"1min":20.439,"5min":448.69200000000001,"15min":448.69200000000001},"last":16.402000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":17.667000000000002,"5min":18.959,"15min":24.385000000000002},"min":{"1min":11.335000000000001,"5min":7.3049999999999997,"15min":7.3049999999999997},"max":{"1min":39.133000000000003,"5min":91.646000000000001,"15min":177.596},"last":17.513999999999999},{"interface":"front","average":{"1min":15.132999999999999,"5min":18.663,"15min":25.454999999999998},"min":{"1min":6.891,"5min":6.891,"15min":6.891},"max":{"1min":35.683,"5min":101.158,"15min":231.03800000000001},"last":21.363}]}]},{"osd":4,"up_from":20,"seq":85899346009,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":8.0920000000000005,"5min":8.8539999999999992,"15min":11.156000000000001},"min":{"1min":5.0110000000000001,"5min":4.5439999999999996,"15min":4.5439999999999996},"max":{"1min":11.35,"5min":35.052,"15min":35.817999999999998},"last":5.9390000000000001},{"interface":"front","average":{"1min":7.8849999999999998,"5min":9.2420000000000009,"15min":11.247},"min":{"1min":4.6500000000000004,"5min":4.6500000000000004,"15min":4.6500000000000004},"max":{"1min":16.132999999999999,"5min":35.091999999999999,"15min":35.091999999999999},"last":9.4570000000000007}]},{"osd":1,"last update":"Fri Aug 30 16:05:29 2024","interfaces":[{"interface":"back","average":{"1min":12.667,"5min":12.869,"15min":11.863},"min":{"1min":7.1399999999999997,"5min":5.6879999999999997,"15min":4.7000000000000002},"max":{"1min":24.204000000000001,"5min":34.347999999999999,"15min":34.347999999999999},"last":11.579000000000001},{"interface":"front","average":{"1min":10.877000000000001,"5min":11.917,"15min":12.324},"min":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"max":{"1min":26.788,"5min":35.481000000000002,"15min":35.481000000000002},"last":7.4669999999999996}]},{"osd":2,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":9.6660000000000004,"5min":9.5470000000000006,"15min":9.9280000000000008},"min":{"1min":5.4930000000000003,"5min":5.4279999999999999,"15min":5.4279999999999999},"max":{"1min":30.666,"5min":30.666,"15min":30.666},"last":12.994},{"interface":"front","average":{"1min":11.707000000000001,"5min":12.987,"15min":14.148999999999999},"min":{"1min":7.0629999999999997,"5min":7.0629999999999997,"15min":7.0629999999999997},"max":{"1min":21.355,"5min":29.079999999999998,"15min":30.629000000000001},"last":14.941000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":11.263,"5min":11.942,"15min":12.234999999999999},"min":{"1min":7.476,"5min":5.8010000000000002,"15min":5.8010000000000002},"max":{"1min":28.315999999999999,"5min":29.715,"15min":29.715},"last":15.586},{"interface":"front","average":{"1min":13.385,"5min":14.148999999999999,"15min":15.195},"min":{"1min":7.8220000000000001,"5min":7.5830000000000002,"15min":7.5830000000000002},"max":{"1min":27.321999999999999,"5min":27.321999999999999,"15min":31.73},"last":9.1039999999999992}]},{"osd":5,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":14.595000000000001,"5min":13.505000000000001,"15min":11.943},"min":{"1min":9.1199999999999992,"5min":6.8639999999999999,"15min":6.5010000000000003},"max":{"1min":31.565000000000001,"5min":42.143000000000001,"15min":42.143000000000001},"last":10.592000000000001},{"interface":"front","average":{"1min":12.859,"5min":13.661,"15min":14.194000000000001},"min":{"1min":6.5579999999999998,"5min":6.5579999999999998,"15min":6.5579999999999998},"max":{"1min":29.210999999999999,"5min":43.917000000000002,"15min":43.917000000000002},"last":13.821}]},{"osd":6,"last update":"Fri Aug 30 16:03:23 2024","interfaces":[{"interface":"back","average":{"1min":13.481,"5min":14.797000000000001,"15min":15.696999999999999},"min":{"1min":8.0359999999999996,"5min":6.5289999999999999,"15min":6.5289999999999999},"max":{"1min":20.704000000000001,"5min":37.215000000000003,"15min":37.215000000000003},"last":6.8600000000000003},{"interface":"front","average":{"1min":16.030999999999999,"5min":16.199000000000002,"15min":17.021999999999998},"min":{"1min":9.8300000000000001,"5min":8.7870000000000008,"15min":8.7870000000000008},"max":{"1min":24.986999999999998,"5min":37.753999999999998,"15min":37.753999999999998},"last":11.07}]},{"osd":7,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":14.657,"15min":15.507999999999999},"min":{"1min":7.9950000000000001,"5min":7.7130000000000001,"15min":7.7130000000000001},"max":{"1min":21.992999999999999,"5min":33.314,"15min":34.551000000000002},"last":10.106999999999999},{"interface":"front","average":{"1min":12.443,"5min":15.465,"15min":16.873999999999999},"min":{"1min":6.375,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":18.116,"5min":35.701999999999998,"15min":35.701999999999998},"last":14.300000000000001}]}]},{"osd":2,"up_from":16,"seq":68719476823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:17 2024","interfaces":[{"interface":"back","average":{"1min":10.505000000000001,"5min":9.4960000000000004,"15min":9.6129999999999995},"min":{"1min":4.1200000000000001,"5min":4.1200000000000001,"15min":3.6709999999999998},"max":{"1min":25.526,"5min":29.303000000000001,"15min":30.366},"last":16.681999999999999},{"interface":"front","average":{"1min":11.006,"5min":9.6720000000000006,"15min":10.569000000000001},"min":{"1min":5.4009999999999998,"5min":4.5839999999999996,"15min":3.4900000000000002},"max":{"1min":25.300000000000001,"5min":27.916,"15min":45.604999999999997},"last":36.712000000000003}]},{"osd":1,"last update":"Fri Aug 30 16:05:33 2024","interfaces":[{"interface":"back","average":{"1min":12.826000000000001,"5min":13.023999999999999,"15min":13.119999999999999},"min":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":41.786999999999999,"5min":41.786999999999999,"15min":46.460000000000001},"last":32.917999999999999},{"interface":"front","average":{"1min":15.141,"5min":11.379,"15min":12.558999999999999},"min":{"1min":6.3650000000000002,"5min":5.944,"15min":5.944},"max":{"1min":67.332999999999998,"5min":67.332999999999998,"15min":67.332999999999998},"last":24.684000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":13.191000000000001,"5min":13.573,"15min":16.579000000000001},"min":{"1min":5.7850000000000001,"5min":5.5179999999999998,"15min":5.5179999999999998},"max":{"1min":31.547999999999998,"5min":71.475999999999999,"15min":71.475999999999999},"last":27.98},{"interface":"front","average":{"1min":14.112,"5min":13.813000000000001,"15min":18.425000000000001},"min":{"1min":6.218,"5min":6.218,"15min":6.218},"max":{"1min":40.046999999999997,"5min":40.046999999999997,"15min":50.564},"last":17.902999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":15.548999999999999,"5min":14.409000000000001,"15min":17.457000000000001},"min":{"1min":8.4000000000000004,"5min":7.0970000000000004,"15min":7.0970000000000004},"max":{"1min":28.899000000000001,"5min":33.892000000000003,"15min":48.673999999999999},"last":19.010999999999999},{"interface":"front","average":{"1min":13.239000000000001,"5min":14.494,"15min":15.609999999999999},"min":{"1min":7.1879999999999997,"5min":6.7640000000000002,"15min":6.7640000000000002},"max":{"1min":27.448,"5min":67.054000000000002,"15min":67.054000000000002},"last":22.93}]},{"osd":5,"last update":"Fri Aug 30 16:05:17 2024","interfaces":[{"interface":"back","average":{"1min":13.17,"5min":15.824,"15min":16.684999999999999},"min":{"1min":7.976,"5min":7.0259999999999998,"15min":4.7199999999999998},"max":{"1min":22.773,"5min":69.411000000000001,"15min":69.411000000000001},"last":34.484999999999999},{"interface":"front","average":{"1min":14.057,"5min":16.693000000000001,"15min":18.975000000000001},"min":{"1min":9.5449999999999999,"5min":8.6910000000000007,"15min":5.7460000000000004},"max":{"1min":24.228999999999999,"5min":56.152000000000001,"15min":56.917999999999999},"last":20.838999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:27 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":16.623000000000001,"15min":18.198},"min":{"1min":10.302,"5min":7.4690000000000003,"15min":7.4690000000000003},"max":{"1min":23.317,"5min":72.75,"15min":72.75},"last":19.917000000000002},{"interface":"front","average":{"1min":19.774000000000001,"5min":18.396999999999998,"15min":20.119},"min":{"1min":9.8879999999999999,"5min":8.1579999999999995,"15min":8.1579999999999995},"max":{"1min":70.462999999999994,"5min":74.119,"15min":74.119},"last":26.353000000000002}]},{"osd":7,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":13.157,"5min":17.632000000000001,"15min":20.654},"min":{"1min":8.3160000000000007,"5min":8.2639999999999993,"15min":8.2639999999999993},"max":{"1min":21.460000000000001,"5min":137.33000000000001,"15min":137.33000000000001},"last":29.675000000000001},{"interface":"front","average":{"1min":15.618,"5min":19.195,"15min":19.751000000000001},"min":{"1min":9.7880000000000003,"5min":7.2770000000000001,"15min":7.2770000000000001},"max":{"1min":30.722999999999999,"5min":136.66399999999999,"15min":136.66399999999999},"last":31.329999999999998}]}]},{"osd":0,"up_from":15,"seq":64424509526,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":7.3120000000000003,"5min":10.007,"15min":9.7769999999999992},"min":{"1min":4.8090000000000002,"5min":4.7489999999999997,"15min":4.7489999999999997},"max":{"1min":14.276,"5min":40.735999999999997,"15min":40.735999999999997},"last":9.2279999999999998},{"interface":"front","average":{"1min":10.282,"5min":11.349,"15min":11.352},"min":{"1min":5.0069999999999997,"5min":4.383,"15min":4.383},"max":{"1min":25.077999999999999,"5min":42.387,"15min":42.387},"last":11.018000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":10.721,"5min":11.506,"15min":11.308},"min":{"1min":6.3899999999999997,"5min":5.3689999999999998,"15min":3.3370000000000002},"max":{"1min":24.170999999999999,"5min":46.768999999999998,"15min":46.768999999999998},"last":9.8079999999999998},{"interface":"front","average":{"1min":13.676,"5min":13.199999999999999,"15min":14.022},"min":{"1min":6.8860000000000001,"5min":6.2249999999999996,"15min":5.702},"max":{"1min":31.361000000000001,"5min":43.921999999999997,"15min":43.921999999999997},"last":21.521999999999998}]},{"osd":3,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":11.460000000000001,"5min":13.585000000000001,"15min":21.405000000000001},"min":{"1min":7.702,"5min":6.2839999999999998,"15min":6.2839999999999998},"max":{"1min":29.998999999999999,"5min":38.219000000000001,"15min":200.155},"last":11.916},{"interface":"front","average":{"1min":15.455,"5min":16.989000000000001,"15min":26.460999999999999},"min":{"1min":10.507999999999999,"5min":7.0940000000000003,"15min":7.0940000000000003},"max":{"1min":28.146000000000001,"5min":44.229999999999997,"15min":267.03100000000001},"last":20.286000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.647,"5min":13.015000000000001,"15min":15.007},"min":{"1min":6.891,"5min":6.0970000000000004,"15min":6.0970000000000004},"max":{"1min":14.891,"5min":31.611999999999998,"15min":56.200000000000003},"last":8.7560000000000002},{"interface":"front","average":{"1min":13.050000000000001,"5min":15.571,"15min":15.683999999999999},"min":{"1min":9.3930000000000007,"5min":9.298,"15min":8.609},"max":{"1min":16.626999999999999,"5min":46.875999999999998,"15min":48.534999999999997},"last":14.603999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":14.43,"5min":15.682,"15min":16.082999999999998},"min":{"1min":9.8059999999999992,"5min":5.9199999999999999,"15min":5.6920000000000002},"max":{"1min":40.006,"5min":43.308,"15min":43.308},"last":12.824999999999999},{"interface":"front","average":{"1min":16.951000000000001,"5min":17.693000000000001,"15min":17.853999999999999},"min":{"1min":9.9100000000000001,"5min":9.2149999999999999,"15min":4.75},"max":{"1min":49.575000000000003,"5min":49.575000000000003,"15min":49.575000000000003},"last":17.23}]},{"osd":6,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":13.808999999999999,"5min":14.292,"15min":14.385999999999999},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":39.460999999999999,"5min":59.328000000000003,"15min":59.328000000000003},"last":19.507999999999999},{"interface":"front","average":{"1min":19.225000000000001,"5min":18.331,"15min":18.533999999999999},"min":{"1min":10.781000000000001,"5min":10.005000000000001,"15min":10.005000000000001},"max":{"1min":46.237000000000002,"5min":58.567,"15min":58.567},"last":16.015999999999998}]},{"osd":7,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":18.472000000000001,"5min":19.276,"15min":21.135999999999999},"min":{"1min":11.045,"5min":10.632999999999999,"15min":9.1180000000000003},"max":{"1min":57.804000000000002,"5min":57.804000000000002,"15min":59.515999999999998},"last":22.613},{"interface":"front","average":{"1min":11.272,"5min":13.384,"15min":14.976000000000001},"min":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":7.0590000000000002},"max":{"1min":14.231999999999999,"5min":37.113,"15min":58.807000000000002},"last":17.939}]}]},{"osd":5,"up_from":13,"seq":55834574934,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":658493,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:20 2024","interfaces":[{"interface":"back","average":{"1min":6.2510000000000003,"5min":8.2520000000000007,"15min":12.516999999999999},"min":{"1min":4.0300000000000002,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":13.52,"5min":53.558999999999997,"15min":111.313},"last":4.9630000000000001},{"interface":"front","average":{"1min":8.1809999999999992,"5min":9.4659999999999993,"15min":13.936},"min":{"1min":4.266,"5min":4.266,"15min":4.266},"max":{"1min":14.064,"5min":52.691000000000003,"15min":107.937},"last":13.064}]},{"osd":1,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":9.3030000000000008,"5min":10.489000000000001,"15min":12.541},"min":{"1min":5.4539999999999997,"5min":4.657,"15min":4.657},"max":{"1min":16.811,"5min":34.545999999999999,"15min":46.988},"last":18.606000000000002},{"interface":"front","average":{"1min":10.688000000000001,"5min":12.9,"15min":15.188000000000001},"min":{"1min":7.5179999999999998,"5min":6.4710000000000001,"15min":6.4710000000000001},"max":{"1min":15.455,"5min":33.203000000000003,"15min":47.661000000000001},"last":13.631}]},{"osd":2,"last update":"Fri Aug 30 16:05:20 2024","interfaces":[{"interface":"back","average":{"1min":13.909000000000001,"5min":15.346,"15min":17.995000000000001},"min":{"1min":8.5630000000000006,"5min":6.9569999999999999,"15min":4.7220000000000004},"max":{"1min":31.667000000000002,"5min":41.549999999999997,"15min":84.317999999999998},"last":16.109000000000002},{"interface":"front","average":{"1min":11.781000000000001,"5min":13.901,"15min":16.207000000000001},"min":{"1min":6.9829999999999997,"5min":6.2770000000000001,"15min":6.0279999999999996},"max":{"1min":29.847999999999999,"5min":43.314999999999998,"15min":99.884},"last":19.725999999999999}]},{"osd":3,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":12.699,"5min":16.114000000000001,"15min":16.768999999999998},"min":{"1min":9.6899999999999995,"5min":6.2149999999999999,"15min":6.2149999999999999},"max":{"1min":18.687000000000001,"5min":117.346,"15min":117.346},"last":17.943000000000001},{"interface":"front","average":{"1min":12.143000000000001,"5min":14.827999999999999,"15min":15.944000000000001},"min":{"1min":6.5270000000000001,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":19.370999999999999,"5min":119.23099999999999,"15min":119.23099999999999},"last":21.300000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":12.651999999999999,"5min":15.768000000000001,"15min":17.32},"min":{"1min":6.5599999999999996,"5min":6.5599999999999996,"15min":6.5599999999999996},"max":{"1min":20.619,"5min":33.582000000000001,"15min":38.095999999999997},"last":20.27},{"interface":"front","average":{"1min":12.183999999999999,"5min":14.702,"15min":16.603999999999999},"min":{"1min":6.71,"5min":6.71,"15min":6.71},"max":{"1min":20.678999999999998,"5min":31.367000000000001,"15min":40.649000000000001},"last":23.041}]},{"osd":6,"last update":"Fri Aug 30 16:03:26 2024","interfaces":[{"interface":"back","average":{"1min":9.516,"5min":10.686,"15min":11.025},"min":{"1min":6.7809999999999997,"5min":6.367,"15min":6.367},"max":{"1min":15.27,"5min":26.344999999999999,"15min":26.344999999999999},"last":10.279999999999999},{"interface":"front","average":{"1min":19.596,"5min":17.696999999999999,"15min":17.396000000000001},"min":{"1min":11.433,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":58.052999999999997,"5min":58.052999999999997,"15min":58.052999999999997},"last":15.548999999999999}]},{"osd":7,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":13.638,"5min":15.441000000000001,"15min":17.472999999999999},"min":{"1min":7.8490000000000002,"5min":7.8490000000000002,"15min":7.8490000000000002},"max":{"1min":25.056999999999999,"5min":38.180999999999997,"15min":43.604999999999997},"last":24.192},{"interface":"front","average":{"1min":9.7629999999999999,"5min":11.205,"15min":16.343},"min":{"1min":5.3579999999999997,"5min":5.3579999999999997,"15min":5.3579999999999997},"max":{"1min":22.509,"5min":31.547999999999998,"15min":88.278000000000006},"last":8.6549999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:06:05.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:06:05.592 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:06:05.593 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:06:05.604 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-08-30T16:06:03.938249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1591398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222156,"kb_used_data":5480,"kb_used_omap":15,"kb_used_meta":216560,"kb_avail":754752564,"statfs":{"total":773094113280,"available":772866625536,"internally_reserved":0,"allocated":5611520,"data_stored":2880942,"data_compressed":1558,"data_compressed_allocated":49152,"data_compressed_original":786432,"omap_allocated":15591,"internal_metadata":221758233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.751840"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.467046+0000","last_change":"2024-08-30T16:04:11.379068+0000","last_active":"2024-08-30T16:05:57.467046+0000","last_peered":"2024-08-30T16:05:57.467046+0000","last_clean":"2024-08-30T16:05:57.467046+0000","last_became_active":"2024-08-30T16:04:11.370524+0000","last_became_peered":"2024-08-30T16:04:11.370524+0000","last_unstale":"2024-08-30T16:05:57.467046+0000","last_undegraded":"2024-08-30T16:05:57.467046+0000","last_fullsized":"2024-08-30T16:05:57.467046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:58:06.441243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.022669+0000","last_change":"2024-08-30T16:04:38.088583+0000","last_active":"2024-08-30T16:05:58.022669+0000","last_peered":"2024-08-30T16:05:58.022669+0000","last_clean":"2024-08-30T16:05:58.022669+0000","last_became_active":"2024-08-30T16:04:38.082821+0000","last_became_peered":"2024-08-30T16:04:38.082821+0000","last_unstale":"2024-08-30T16:05:58.022669+0000","last_undegraded":"2024-08-30T16:05:58.022669+0000","last_fullsized":"2024-08-30T16:05:58.022669+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:40:53.077427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.520451+0000","last_change":"2024-08-30T16:05:38.369602+0000","last_active":"2024-08-30T16:05:57.520451+0000","last_peered":"2024-08-30T16:05:57.520451+0000","last_clean":"2024-08-30T16:05:57.520451+0000","last_became_active":"2024-08-30T16:05:38.361240+0000","last_became_peered":"2024-08-30T16:05:38.361240+0000","last_unstale":"2024-08-30T16:05:57.520451+0000","last_undegraded":"2024-08-30T16:05:57.520451+0000","last_fullsized":"2024-08-30T16:05:57.520451+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:02:07.953195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.259961+0000","last_change":"2024-08-30T16:05:39.779631+0000","last_active":"2024-08-30T16:05:57.259961+0000","last_peered":"2024-08-30T16:05:57.259961+0000","last_clean":"2024-08-30T16:05:57.259961+0000","last_became_active":"2024-08-30T16:05:39.773479+0000","last_became_peered":"2024-08-30T16:05:39.773479+0000","last_unstale":"2024-08-30T16:05:57.259961+0000","last_undegraded":"2024-08-30T16:05:57.259961+0000","last_fullsized":"2024-08-30T16:05:57.259961+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:29:29.455454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.255363+0000","last_change":"2024-08-30T16:05:40.031358+0000","last_active":"2024-08-30T16:05:57.255363+0000","last_peered":"2024-08-30T16:05:57.255363+0000","last_clean":"2024-08-30T16:05:57.255363+0000","last_became_active":"2024-08-30T16:05:39.776453+0000","last_became_peered":"2024-08-30T16:05:39.776453+0000","last_unstale":"2024-08-30T16:05:57.255363+0000","last_undegraded":"2024-08-30T16:05:57.255363+0000","last_fullsized":"2024-08-30T16:05:57.255363+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:06:14.092608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.462397+0000","last_change":"2024-08-30T16:05:38.365381+0000","last_active":"2024-08-30T16:05:57.462397+0000","last_peered":"2024-08-30T16:05:57.462397+0000","last_clean":"2024-08-30T16:05:57.462397+0000","last_became_active":"2024-08-30T16:05:38.358584+0000","last_became_peered":"2024-08-30T16:05:38.358584+0000","last_unstale":"2024-08-30T16:05:57.462397+0000","last_undegraded":"2024-08-30T16:05:57.462397+0000","last_fullsized":"2024-08-30T16:05:57.462397+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:28:34.896593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":141,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.257472+0000","last_change":"2024-08-30T16:05:38.364964+0000","last_active":"2024-08-30T16:05:57.257472+0000","last_peered":"2024-08-30T16:05:57.257472+0000","last_clean":"2024-08-30T16:05:57.257472+0000","last_became_active":"2024-08-30T16:05:38.358598+0000","last_became_peered":"2024-08-30T16:05:38.358598+0000","last_unstale":"2024-08-30T16:05:57.257472+0000","last_undegraded":"2024-08-30T16:05:57.257472+0000","last_fullsized":"2024-08-30T16:05:57.257472+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T03:36:43.310274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":61,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.904323+0000","last_change":"2024-08-30T16:04:08.039936+0000","last_active":"2024-08-30T16:05:58.904323+0000","last_peered":"2024-08-30T16:05:58.904323+0000","last_clean":"2024-08-30T16:05:58.904323+0000","last_became_active":"2024-08-30T16:04:07.760364+0000","last_became_peered":"2024-08-30T16:04:07.760364+0000","last_unstale":"2024-08-30T16:05:58.904323+0000","last_undegraded":"2024-08-30T16:05:58.904323+0000","last_fullsized":"2024-08-30T16:05:58.904323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:13:37.016725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":160,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.024190+0000","last_change":"2024-08-30T16:05:38.553500+0000","last_active":"2024-08-30T16:05:58.024190+0000","last_peered":"2024-08-30T16:05:58.024190+0000","last_clean":"2024-08-30T16:05:58.024190+0000","last_became_active":"2024-08-30T16:05:38.360837+0000","last_became_peered":"2024-08-30T16:05:38.360837+0000","last_unstale":"2024-08-30T16:05:58.024190+0000","last_undegraded":"2024-08-30T16:05:58.024190+0000","last_fullsized":"2024-08-30T16:05:58.024190+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:15:13.872133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1591360,"data_compressed":3116,"data_compressed_allocated":98304,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280602,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27732,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344108,"statfs":{"total":96636764160,"available":96608366592,"internally_reserved":0,"allocated":724992,"data_stored":658493,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.896000000000001,"5min":11.542,"15min":13.461},"min":{"1min":5.5519999999999996,"5min":4.8049999999999997,"15min":4.8049999999999997},"max":{"1min":25.129999999999999,"5min":51.027000000000001,"15min":57.177999999999997},"last":6.0860000000000003},{"interface":"front","average":{"1min":9.5020000000000007,"5min":11.342000000000001,"15min":12.673999999999999},"min":{"1min":4.5759999999999996,"5min":4.5759999999999996,"15min":4.5759999999999996},"max":{"1min":16.521000000000001,"5min":54.682000000000002,"15min":58.024999999999999},"last":8.3930000000000007}]},{"osd":2,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.157,"5min":10.984999999999999,"15min":10.614000000000001},"min":{"1min":6.0880000000000001,"5min":5.5780000000000003,"15min":5.5780000000000003},"max":{"1min":15.81,"5min":73.399000000000001,"15min":73.399000000000001},"last":8.0540000000000003},{"interface":"front","average":{"1min":13.478,"5min":14.470000000000001,"15min":14.875},"min":{"1min":6.5640000000000001,"5min":6.0049999999999999,"15min":6.0049999999999999},"max":{"1min":30.704999999999998,"5min":75.120000000000005,"15min":75.120000000000005},"last":10.186999999999999}]},{"osd":3,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":13.753,"5min":15.361000000000001,"15min":16.896000000000001},"min":{"1min":9.1199999999999992,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":29.506,"5min":66.869,"15min":66.869},"last":10.727},{"interface":"front","average":{"1min":10.523999999999999,"5min":13.089,"15min":14.315},"min":{"1min":5.8959999999999999,"5min":5.8959999999999999,"15min":5.8959999999999999},"max":{"1min":16.196000000000002,"5min":104.38200000000001,"15min":104.38200000000001},"last":9.6280000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":11.037000000000001,"5min":14.484999999999999,"15min":14.68},"min":{"1min":7.5590000000000002,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":18.202999999999999,"5min":66.340999999999994,"15min":66.340999999999994},"last":13.754},{"interface":"front","average":{"1min":9.5540000000000003,"5min":12.513999999999999,"15min":12.881},"min":{"1min":6.774,"5min":6.774,"15min":6.774},"max":{"1min":14.335000000000001,"5min":29.818999999999999,"15min":29.818999999999999},"last":13.16}]},{"osd":5,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":13.954000000000001,"5min":15.705,"15min":16.510999999999999},"min":{"1min":7.21,"5min":7.21,"15min":7.21},"max":{"1min":31.545000000000002,"5min":55.784999999999997,"15min":55.784999999999997},"last":11.273999999999999},{"interface":"front","average":{"1min":11.759,"5min":17.265999999999998,"15min":18.946000000000002},"min":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"max":{"1min":16.829999999999998,"5min":68.004999999999995,"15min":68.004999999999995},"last":16.846}]},{"osd":6,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":15.708,"5min":16.609000000000002,"15min":17.382000000000001},"min":{"1min":10.180999999999999,"5min":9.1929999999999996,"15min":9.1929999999999996},"max":{"1min":27.353999999999999,"5min":58.201999999999998,"15min":58.201999999999998},"last":14.301},{"interface":"front","average":{"1min":16.324999999999999,"5min":16.748000000000001,"15min":17.001000000000001},"min":{"1min":9.6980000000000004,"5min":8.8390000000000004,"15min":8.8390000000000004},"max":{"1min":26.257999999999999,"5min":37.253,"15min":37.253},"last":15.581}]},{"osd":7,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":16.931000000000001,"5min":17.739999999999998,"15min":19.425999999999998},"min":{"1min":10.352,"5min":8.2050000000000001,"15min":8.2050000000000001},"max":{"1min":34.345999999999997,"5min":68.950000000000003,"15min":68.950000000000003},"last":16.158000000000001},{"interface":"front","average":{"1min":13.039999999999999,"5min":16.266999999999999,"15min":18.443999999999999},"min":{"1min":9.4529999999999994,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":17.625,"5min":36.186,"15min":59.079000000000001},"last":17.533999999999999}]}]},{"osd":7,"up_from":21,"seq":90194313306,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":7.9219999999999997,"5min":8.6739999999999995,"15min":8.8670000000000009},"min":{"1min":4.3789999999999996,"5min":4.3789999999999996,"15min":4.3789999999999996},"max":{"1min":20.73,"5min":25.759,"15min":30.600999999999999},"last":14.135999999999999},{"interface":"front","average":{"1min":7.1589999999999998,"5min":8.8819999999999997,"15min":9.0419999999999998},"min":{"1min":4.8460000000000001,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":12.57,"5min":28.077000000000002,"15min":28.077000000000002},"last":6.0970000000000004}]},{"osd":1,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":11.804,"5min":12.818,"15min":11.496},"min":{"1min":7.0590000000000002,"5min":5.7009999999999996,"15min":5.7009999999999996},"max":{"1min":18.989999999999998,"5min":83.581000000000003,"15min":83.581000000000003},"last":14.663},{"interface":"front","average":{"1min":9.2690000000000001,"5min":11.781000000000001,"15min":12.198},"min":{"1min":5.6230000000000002,"5min":5.6230000000000002,"15min":5.6230000000000002},"max":{"1min":19.524999999999999,"5min":81.722999999999999,"15min":81.722999999999999},"last":7.5270000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":14.83,"5min":13.890000000000001,"15min":13.689},"min":{"1min":8.5039999999999996,"5min":5.4850000000000003,"15min":5.4850000000000003},"max":{"1min":25.885999999999999,"5min":42.506,"15min":42.506},"last":18.111999999999998},{"interface":"front","average":{"1min":13.273999999999999,"5min":13.69,"15min":16.242000000000001},"min":{"1min":6.8029999999999999,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":24.309000000000001,"5min":43.484999999999999,"15min":43.484999999999999},"last":15.276}]},{"osd":3,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":15.515000000000001,"5min":15.986000000000001,"15min":16.391999999999999},"min":{"1min":7.9299999999999997,"5min":7.9299999999999997,"15min":7.9299999999999997},"max":{"1min":54.076999999999998,"5min":65.739999999999995,"15min":65.739999999999995},"last":8.4290000000000003},{"interface":"front","average":{"1min":12.406000000000001,"5min":13.285,"15min":12.871},"min":{"1min":5.6070000000000002,"5min":5.6070000000000002,"15min":5.6070000000000002},"max":{"1min":55.978000000000002,"5min":64.980000000000004,"15min":64.980000000000004},"last":17.135000000000002}]},{"osd":4,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":13.824,"15min":12.827},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.4880000000000004},"max":{"1min":23.213999999999999,"5min":41.631,"15min":41.631},"last":9.6419999999999995},{"interface":"front","average":{"1min":14.590999999999999,"5min":15.433,"15min":17.003},"min":{"1min":9.2789999999999999,"5min":8.8729999999999993,"15min":8.8729999999999993},"max":{"1min":25.149999999999999,"5min":32.526000000000003,"15min":37.393000000000001},"last":12.411}]},{"osd":5,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.68,"5min":12.377000000000001,"15min":11.679},"min":{"1min":6.8259999999999996,"5min":6.2309999999999999,"15min":6.2309999999999999},"max":{"1min":37.372999999999998,"5min":37.372999999999998,"15min":37.372999999999998},"last":16.436},{"interface":"front","average":{"1min":16.558,"5min":15.769,"15min":16.600000000000001},"min":{"1min":9.0429999999999993,"5min":8.2870000000000008,"15min":8.2870000000000008},"max":{"1min":38.484000000000002,"5min":38.618000000000002,"15min":38.618000000000002},"last":10.513}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.326000000000001,"5min":13.771000000000001,"15min":13.208},"min":{"1min":8.0060000000000002,"5min":6.3230000000000004,"15min":6.3230000000000004},"max":{"1min":19.231000000000002,"5min":42.829999999999998,"15min":42.829999999999998},"last":11.218},{"interface":"front","average":{"1min":15.842000000000001,"5min":17.606000000000002,"15min":18.088999999999999},"min":{"1min":11.712,"5min":10.749000000000001,"15min":10.749000000000001},"max":{"1min":22.695,"5min":44.491999999999997,"15min":44.491999999999997},"last":13.292}]}]},{"osd":6,"up_from":105,"seq":450971566091,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":680,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":696320,"data_stored":260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":28176046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.835000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.928000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9849999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.256}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.364000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":696320,"data_stored":260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":9.4120000000000008,"5min":9.5199999999999996,"15min":10.925000000000001},"min":{"1min":5.0830000000000002,"5min":4.3810000000000002,"15min":4.29},"max":{"1min":18.713000000000001,"5min":46.146999999999998,"15min":50.500999999999998},"last":4.7599999999999998},{"interface":"front","average":{"1min":9.7110000000000003,"5min":10.157,"15min":13.363},"min":{"1min":5.9279999999999999,"5min":5.1360000000000001,"15min":5.1360000000000001},"max":{"1min":20.637,"5min":36.720999999999997,"15min":51.965000000000003},"last":9.9149999999999991}]},{"osd":1,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":12.821999999999999,"5min":13.605,"15min":17.742999999999999},"min":{"1min":7.7549999999999999,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":20.864999999999998,"5min":44.862000000000002,"15min":137.56700000000001},"last":7.7279999999999998},{"interface":"front","average":{"1min":10.217000000000001,"5min":12.255000000000001,"15min":15.741},"min":{"1min":6.7839999999999998,"5min":5.0060000000000002,"15min":5.0060000000000002},"max":{"1min":27.032,"5min":47.530999999999999,"15min":136.321},"last":13.763}]},{"osd":2,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":11.897,"5min":12.744999999999999,"15min":15.519},"min":{"1min":5.9109999999999996,"5min":5.9109999999999996,"15min":5.9109999999999996},"max":{"1min":26.527000000000001,"5min":43.869999999999997,"15min":73.831000000000003},"last":11.119999999999999},{"interface":"front","average":{"1min":11.170999999999999,"5min":11.682,"15min":14.923},"min":{"1min":6.984,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":16.887,"5min":26.074999999999999,"15min":71.831999999999994},"last":20.190000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":14.353,"15min":26.016999999999999},"min":{"1min":7.2350000000000003,"5min":6.9199999999999999,"15min":6.9199999999999999},"max":{"1min":25.645,"5min":43.235999999999997,"15min":342.34500000000003},"last":8.9779999999999998},{"interface":"front","average":{"1min":12.157,"5min":14.849,"15min":19.047000000000001},"min":{"1min":7.1050000000000004,"5min":6.7859999999999996,"15min":6.7859999999999996},"max":{"1min":23.939,"5min":63.633000000000003,"15min":127.857},"last":18.582000000000001}]},{"osd":5,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":10.967000000000001,"5min":14.295,"15min":15.500999999999999},"min":{"1min":7.1200000000000001,"5min":6.5,"15min":6.5},"max":{"1min":16.876000000000001,"5min":58.715000000000003,"15min":58.715000000000003},"last":11.959},{"interface":"front","average":{"1min":14.762,"5min":16.952000000000002,"15min":18.649000000000001},"min":{"1min":9.1549999999999994,"5min":7.5869999999999997,"15min":7.2370000000000001},"max":{"1min":29.803999999999998,"5min":58.212000000000003,"15min":58.212000000000003},"last":15.076000000000001}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.295000000000002,"5min":23.492000000000001,"15min":30.053999999999998},"min":{"1min":11.096,"5min":8.6630000000000003,"15min":8.6630000000000003},"max":{"1min":25.298999999999999,"5min":388.96100000000001,"15min":388.96100000000001},"last":13.083},{"interface":"front","average":{"1min":13.961,"5min":24.18,"15min":32.835999999999999},"min":{"1min":9.4079999999999995,"5min":8.5519999999999996,"15min":8.5519999999999996},"max":{"1min":20.439,"5min":448.69200000000001,"15min":448.69200000000001},"last":16.402000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":17.667000000000002,"5min":18.959,"15min":24.385000000000002},"min":{"1min":11.335000000000001,"5min":7.3049999999999997,"15min":7.3049999999999997},"max":{"1min":39.133000000000003,"5min":91.646000000000001,"15min":177.596},"last":17.513999999999999},{"interface":"front","average":{"1min":15.132999999999999,"5min":18.663,"15min":25.454999999999998},"min":{"1min":6.891,"5min":6.891,"15min":6.891},"max":{"1min":35.683,"5min":101.158,"15min":231.03800000000001},"last":21.363}]}]},{"osd":4,"up_from":20,"seq":85899346009,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":8.0920000000000005,"5min":8.8539999999999992,"15min":11.156000000000001},"min":{"1min":5.0110000000000001,"5min":4.5439999999999996,"15min":4.5439999999999996},"max":{"1min":11.35,"5min":35.052,"15min":35.817999999999998},"last":5.9390000000000001},{"interface":"front","average":{"1min":7.8849999999999998,"5min":9.2420000000000009,"15min":11.247},"min":{"1min":4.6500000000000004,"5min":4.6500000000000004,"15min":4.6500000000000004},"max":{"1min":16.132999999999999,"5min":35.091999999999999,"15min":35.091999999999999},"last":9.4570000000000007}]},{"osd":1,"last update":"Fri Aug 30 16:05:29 2024","interfaces":[{"interface":"back","average":{"1min":12.667,"5min":12.869,"15min":11.863},"min":{"1min":7.1399999999999997,"5min":5.6879999999999997,"15min":4.7000000000000002},"max":{"1min":24.204000000000001,"5min":34.347999999999999,"15min":34.347999999999999},"last":11.579000000000001},{"interface":"front","average":{"1min":10.877000000000001,"5min":11.917,"15min":12.324},"min":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"max":{"1min":26.788,"5min":35.481000000000002,"15min":35.481000000000002},"last":7.4669999999999996}]},{"osd":2,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":9.6660000000000004,"5min":9.5470000000000006,"15min":9.9280000000000008},"min":{"1min":5.4930000000000003,"5min":5.4279999999999999,"15min":5.4279999999999999},"max":{"1min":30.666,"5min":30.666,"15min":30.666},"last":12.994},{"interface":"front","average":{"1min":11.707000000000001,"5min":12.987,"15min":14.148999999999999},"min":{"1min":7.0629999999999997,"5min":7.0629999999999997,"15min":7.0629999999999997},"max":{"1min":21.355,"5min":29.079999999999998,"15min":30.629000000000001},"last":14.941000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":11.263,"5min":11.942,"15min":12.234999999999999},"min":{"1min":7.476,"5min":5.8010000000000002,"15min":5.8010000000000002},"max":{"1min":28.315999999999999,"5min":29.715,"15min":29.715},"last":15.586},{"interface":"front","average":{"1min":13.385,"5min":14.148999999999999,"15min":15.195},"min":{"1min":7.8220000000000001,"5min":7.5830000000000002,"15min":7.5830000000000002},"max":{"1min":27.321999999999999,"5min":27.321999999999999,"15min":31.73},"last":9.1039999999999992}]},{"osd":5,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":14.595000000000001,"5min":13.505000000000001,"15min":11.943},"min":{"1min":9.1199999999999992,"5min":6.8639999999999999,"15min":6.5010000000000003},"max":{"1min":31.565000000000001,"5min":42.143000000000001,"15min":42.143000000000001},"last":10.592000000000001},{"interface":"front","average":{"1min":12.859,"5min":13.661,"15min":14.194000000000001},"min":{"1min":6.5579999999999998,"5min":6.5579999999999998,"15min":6.5579999999999998},"max":{"1min":29.210999999999999,"5min":43.917000000000002,"15min":43.917000000000002},"last":13.821}]},{"osd":6,"last update":"Fri Aug 30 16:03:23 2024","interfaces":[{"interface":"back","average":{"1min":13.481,"5min":14.797000000000001,"15min":15.696999999999999},"min":{"1min":8.0359999999999996,"5min":6.5289999999999999,"15min":6.5289999999999999},"max":{"1min":20.704000000000001,"5min":37.215000000000003,"15min":37.215000000000003},"last":6.8600000000000003},{"interface":"front","average":{"1min":16.030999999999999,"5min":16.199000000000002,"15min":17.021999999999998},"min":{"1min":9.8300000000000001,"5min":8.7870000000000008,"15min":8.7870000000000008},"max":{"1min":24.986999999999998,"5min":37.753999999999998,"15min":37.753999999999998},"last":11.07}]},{"osd":7,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":14.657,"15min":15.507999999999999},"min":{"1min":7.9950000000000001,"5min":7.7130000000000001,"15min":7.7130000000000001},"max":{"1min":21.992999999999999,"5min":33.314,"15min":34.551000000000002},"last":10.106999999999999},{"interface":"front","average":{"1min":12.443,"5min":15.465,"15min":16.873999999999999},"min":{"1min":6.375,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":18.116,"5min":35.701999999999998,"15min":35.701999999999998},"last":14.300000000000001}]}]},{"osd":2,"up_from":16,"seq":68719476823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:17 2024","interfaces":[{"interface":"back","average":{"1min":10.505000000000001,"5min":9.4960000000000004,"15min":9.6129999999999995},"min":{"1min":4.1200000000000001,"5min":4.1200000000000001,"15min":3.6709999999999998},"max":{"1min":25.526,"5min":29.303000000000001,"15min":30.366},"last":16.681999999999999},{"interface":"front","average":{"1min":11.006,"5min":9.6720000000000006,"15min":10.569000000000001},"min":{"1min":5.4009999999999998,"5min":4.5839999999999996,"15min":3.4900000000000002},"max":{"1min":25.300000000000001,"5min":27.916,"15min":45.604999999999997},"last":36.712000000000003}]},{"osd":1,"last update":"Fri Aug 30 16:05:33 2024","interfaces":[{"interface":"back","average":{"1min":12.826000000000001,"5min":13.023999999999999,"15min":13.119999999999999},"min":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":41.786999999999999,"5min":41.786999999999999,"15min":46.460000000000001},"last":32.917999999999999},{"interface":"front","average":{"1min":15.141,"5min":11.379,"15min":12.558999999999999},"min":{"1min":6.3650000000000002,"5min":5.944,"15min":5.944},"max":{"1min":67.332999999999998,"5min":67.332999999999998,"15min":67.332999999999998},"last":24.684000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":13.191000000000001,"5min":13.573,"15min":16.579000000000001},"min":{"1min":5.7850000000000001,"5min":5.5179999999999998,"15min":5.5179999999999998},"max":{"1min":31.547999999999998,"5min":71.475999999999999,"15min":71.475999999999999},"last":27.98},{"interface":"front","average":{"1min":14.112,"5min":13.813000000000001,"15min":18.425000000000001},"min":{"1min":6.218,"5min":6.218,"15min":6.218},"max":{"1min":40.046999999999997,"5min":40.046999999999997,"15min":50.564},"last":17.902999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":15.548999999999999,"5min":14.409000000000001,"15min":17.457000000000001},"min":{"1min":8.4000000000000004,"5min":7.0970000000000004,"15min":7.0970000000000004},"max":{"1min":28.899000000000001,"5min":33.892000000000003,"15min":48.673999999999999},"last":19.010999999999999},{"interface":"front","average":{"1min":13.239000000000001,"5min":14.494,"15min":15.609999999999999},"min":{"1min":7.1879999999999997,"5min":6.7640000000000002,"15min":6.7640000000000002},"max":{"1min":27.448,"5min":67.054000000000002,"15min":67.054000000000002},"last":22.93}]},{"osd":5,"last update":"Fri Aug 30 16:05:17 2024","interfaces":[{"interface":"back","average":{"1min":13.17,"5min":15.824,"15min":16.684999999999999},"min":{"1min":7.976,"5min":7.0259999999999998,"15min":4.7199999999999998},"max":{"1min":22.773,"5min":69.411000000000001,"15min":69.411000000000001},"last":34.484999999999999},{"interface":"front","average":{"1min":14.057,"5min":16.693000000000001,"15min":18.975000000000001},"min":{"1min":9.5449999999999999,"5min":8.6910000000000007,"15min":5.7460000000000004},"max":{"1min":24.228999999999999,"5min":56.152000000000001,"15min":56.917999999999999},"last":20.838999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:27 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":16.623000000000001,"15min":18.198},"min":{"1min":10.302,"5min":7.4690000000000003,"15min":7.4690000000000003},"max":{"1min":23.317,"5min":72.75,"15min":72.75},"last":19.917000000000002},{"interface":"front","average":{"1min":19.774000000000001,"5min":18.396999999999998,"15min":20.119},"min":{"1min":9.8879999999999999,"5min":8.1579999999999995,"15min":8.1579999999999995},"max":{"1min":70.462999999999994,"5min":74.119,"15min":74.119},"last":26.353000000000002}]},{"osd":7,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":13.157,"5min":17.632000000000001,"15min":20.654},"min":{"1min":8.3160000000000007,"5min":8.2639999999999993,"15min":8.2639999999999993},"max":{"1min":21.460000000000001,"5min":137.33000000000001,"15min":137.33000000000001},"last":29.675000000000001},{"interface":"front","average":{"1min":15.618,"5min":19.195,"15min":19.751000000000001},"min":{"1min":9.7880000000000003,"5min":7.2770000000000001,"15min":7.2770000000000001},"max":{"1min":30.722999999999999,"5min":136.66399999999999,"15min":136.66399999999999},"last":31.329999999999998}]}]},{"osd":0,"up_from":15,"seq":64424509526,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":7.3120000000000003,"5min":10.007,"15min":9.7769999999999992},"min":{"1min":4.8090000000000002,"5min":4.7489999999999997,"15min":4.7489999999999997},"max":{"1min":14.276,"5min":40.735999999999997,"15min":40.735999999999997},"last":9.2279999999999998},{"interface":"front","average":{"1min":10.282,"5min":11.349,"15min":11.352},"min":{"1min":5.0069999999999997,"5min":4.383,"15min":4.383},"max":{"1min":25.077999999999999,"5min":42.387,"15min":42.387},"last":11.018000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":10.721,"5min":11.506,"15min":11.308},"min":{"1min":6.3899999999999997,"5min":5.3689999999999998,"15min":3.3370000000000002},"max":{"1min":24.170999999999999,"5min":46.768999999999998,"15min":46.768999999999998},"last":9.8079999999999998},{"interface":"front","average":{"1min":13.676,"5min":13.199999999999999,"15min":14.022},"min":{"1min":6.8860000000000001,"5min":6.2249999999999996,"15min":5.702},"max":{"1min":31.361000000000001,"5min":43.921999999999997,"15min":43.921999999999997},"last":21.521999999999998}]},{"osd":3,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":11.460000000000001,"5min":13.585000000000001,"15min":21.405000000000001},"min":{"1min":7.702,"5min":6.2839999999999998,"15min":6.2839999999999998},"max":{"1min":29.998999999999999,"5min":38.219000000000001,"15min":200.155},"last":11.916},{"interface":"front","average":{"1min":15.455,"5min":16.989000000000001,"15min":26.460999999999999},"min":{"1min":10.507999999999999,"5min":7.0940000000000003,"15min":7.0940000000000003},"max":{"1min":28.146000000000001,"5min":44.229999999999997,"15min":267.03100000000001},"last":20.286000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.647,"5min":13.015000000000001,"15min":15.007},"min":{"1min":6.891,"5min":6.0970000000000004,"15min":6.0970000000000004},"max":{"1min":14.891,"5min":31.611999999999998,"15min":56.200000000000003},"last":8.7560000000000002},{"interface":"front","average":{"1min":13.050000000000001,"5min":15.571,"15min":15.683999999999999},"min":{"1min":9.3930000000000007,"5min":9.298,"15min":8.609},"max":{"1min":16.626999999999999,"5min":46.875999999999998,"15min":48.534999999999997},"last":14.603999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":14.43,"5min":15.682,"15min":16.082999999999998},"min":{"1min":9.8059999999999992,"5min":5.9199999999999999,"15min":5.6920000000000002},"max":{"1min":40.006,"5min":43.308,"15min":43.308},"last":12.824999999999999},{"interface":"front","average":{"1min":16.951000000000001,"5min":17.693000000000001,"15min":17.853999999999999},"min":{"1min":9.9100000000000001,"5min":9.2149999999999999,"15min":4.75},"max":{"1min":49.575000000000003,"5min":49.575000000000003,"15min":49.575000000000003},"last":17.23}]},{"osd":6,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":13.808999999999999,"5min":14.292,"15min":14.385999999999999},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":39.460999999999999,"5min":59.328000000000003,"15min":59.328000000000003},"last":19.507999999999999},{"interface":"front","average":{"1min":19.225000000000001,"5min":18.331,"15min":18.533999999999999},"min":{"1min":10.781000000000001,"5min":10.005000000000001,"15min":10.005000000000001},"max":{"1min":46.237000000000002,"5min":58.567,"15min":58.567},"last":16.015999999999998}]},{"osd":7,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":18.472000000000001,"5min":19.276,"15min":21.135999999999999},"min":{"1min":11.045,"5min":10.632999999999999,"15min":9.1180000000000003},"max":{"1min":57.804000000000002,"5min":57.804000000000002,"15min":59.515999999999998},"last":22.613},{"interface":"front","average":{"1min":11.272,"5min":13.384,"15min":14.976000000000001},"min":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":7.0590000000000002},"max":{"1min":14.231999999999999,"5min":37.113,"15min":58.807000000000002},"last":17.939}]}]},{"osd":5,"up_from":13,"seq":55834574934,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":658493,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:20 2024","interfaces":[{"interface":"back","average":{"1min":6.2510000000000003,"5min":8.2520000000000007,"15min":12.516999999999999},"min":{"1min":4.0300000000000002,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":13.52,"5min":53.558999999999997,"15min":111.313},"last":4.9630000000000001},{"interface":"front","average":{"1min":8.1809999999999992,"5min":9.4659999999999993,"15min":13.936},"min":{"1min":4.266,"5min":4.266,"15min":4.266},"max":{"1min":14.064,"5min":52.691000000000003,"15min":107.937},"last":13.064}]},{"osd":1,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":9.3030000000000008,"5min":10.489000000000001,"15min":12.541},"min":{"1min":5.4539999999999997,"5min":4.657,"15min":4.657},"max":{"1min":16.811,"5min":34.545999999999999,"15min":46.988},"last":18.606000000000002},{"interface":"front","average":{"1min":10.688000000000001,"5min":12.9,"15min":15.188000000000001},"min":{"1min":7.5179999999999998,"5min":6.4710000000000001,"15min":6.4710000000000001},"max":{"1min":15.455,"5min":33.203000000000003,"15min":47.661000000000001},"last":13.631}]},{"osd":2,"last update":"Fri Aug 30 16:05:20 2024","interfaces":[{"interface":"back","average":{"1min":13.909000000000001,"5min":15.346,"15min":17.995000000000001},"min":{"1min":8.5630000000000006,"5min":6.9569999999999999,"15min":4.7220000000000004},"max":{"1min":31.667000000000002,"5min":41.549999999999997,"15min":84.317999999999998},"last":16.109000000000002},{"interface":"front","average":{"1min":11.781000000000001,"5min":13.901,"15min":16.207000000000001},"min":{"1min":6.9829999999999997,"5min":6.2770000000000001,"15min":6.0279999999999996},"max":{"1min":29.847999999999999,"5min":43.314999999999998,"15min":99.884},"last":19.725999999999999}]},{"osd":3,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":12.699,"5min":16.114000000000001,"15min":16.768999999999998},"min":{"1min":9.6899999999999995,"5min":6.2149999999999999,"15min":6.2149999999999999},"max":{"1min":18.687000000000001,"5min":117.346,"15min":117.346},"last":17.943000000000001},{"interface":"front","average":{"1min":12.143000000000001,"5min":14.827999999999999,"15min":15.944000000000001},"min":{"1min":6.5270000000000001,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":19.370999999999999,"5min":119.23099999999999,"15min":119.23099999999999},"last":21.300000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":12.651999999999999,"5min":15.768000000000001,"15min":17.32},"min":{"1min":6.5599999999999996,"5min":6.5599999999999996,"15min":6.5599999999999996},"max":{"1min":20.619,"5min":33.582000000000001,"15min":38.095999999999997},"last":20.27},{"interface":"front","average":{"1min":12.183999999999999,"5min":14.702,"15min":16.603999999999999},"min":{"1min":6.71,"5min":6.71,"15min":6.71},"max":{"1min":20.678999999999998,"5min":31.367000000000001,"15min":40.649000000000001},"last":23.041}]},{"osd":6,"last update":"Fri Aug 30 16:03:26 2024","interfaces":[{"interface":"back","average":{"1min":9.516,"5min":10.686,"15min":11.025},"min":{"1min":6.7809999999999997,"5min":6.367,"15min":6.367},"max":{"1min":15.27,"5min":26.344999999999999,"15min":26.344999999999999},"last":10.279999999999999},{"interface":"front","average":{"1min":19.596,"5min":17.696999999999999,"15min":17.396000000000001},"min":{"1min":11.433,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":58.052999999999997,"5min":58.052999999999997,"15min":58.052999999999997},"last":15.548999999999999}]},{"osd":7,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":13.638,"5min":15.441000000000001,"15min":17.472999999999999},"min":{"1min":7.8490000000000002,"5min":7.8490000000000002,"15min":7.8490000000000002},"max":{"1min":25.056999999999999,"5min":38.180999999999997,"15min":43.604999999999997},"last":24.192},{"interface":"front","average":{"1min":9.7629999999999999,"5min":11.205,"15min":16.343},"min":{"1min":5.3579999999999997,"5min":5.3579999999999997,"15min":5.3579999999999997},"max":{"1min":22.509,"5min":31.547999999999998,"15min":88.278000000000006},"last":8.6549999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:06:05.606 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-08-30T16:06:05.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-08-30T16:06:06.400 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-30T16:06:06.412 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-30T16:06:06.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:06:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:06:06.953 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:06:06.964 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-08-30T16:06:05.938627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1591398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222156,"kb_used_data":5480,"kb_used_omap":15,"kb_used_meta":216560,"kb_avail":754752564,"statfs":{"total":773094113280,"available":772866625536,"internally_reserved":0,"allocated":5611520,"data_stored":2880942,"data_compressed":1558,"data_compressed_allocated":49152,"data_compressed_original":786432,"omap_allocated":15591,"internal_metadata":221758233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.740013"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.467046+0000","last_change":"2024-08-30T16:04:11.379068+0000","last_active":"2024-08-30T16:05:57.467046+0000","last_peered":"2024-08-30T16:05:57.467046+0000","last_clean":"2024-08-30T16:05:57.467046+0000","last_became_active":"2024-08-30T16:04:11.370524+0000","last_became_peered":"2024-08-30T16:04:11.370524+0000","last_unstale":"2024-08-30T16:05:57.467046+0000","last_undegraded":"2024-08-30T16:05:57.467046+0000","last_fullsized":"2024-08-30T16:05:57.467046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:58:06.441243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.022669+0000","last_change":"2024-08-30T16:04:38.088583+0000","last_active":"2024-08-30T16:05:58.022669+0000","last_peered":"2024-08-30T16:05:58.022669+0000","last_clean":"2024-08-30T16:05:58.022669+0000","last_became_active":"2024-08-30T16:04:38.082821+0000","last_became_peered":"2024-08-30T16:04:38.082821+0000","last_unstale":"2024-08-30T16:05:58.022669+0000","last_undegraded":"2024-08-30T16:05:58.022669+0000","last_fullsized":"2024-08-30T16:05:58.022669+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:40:53.077427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.520451+0000","last_change":"2024-08-30T16:05:38.369602+0000","last_active":"2024-08-30T16:05:57.520451+0000","last_peered":"2024-08-30T16:05:57.520451+0000","last_clean":"2024-08-30T16:05:57.520451+0000","last_became_active":"2024-08-30T16:05:38.361240+0000","last_became_peered":"2024-08-30T16:05:38.361240+0000","last_unstale":"2024-08-30T16:05:57.520451+0000","last_undegraded":"2024-08-30T16:05:57.520451+0000","last_fullsized":"2024-08-30T16:05:57.520451+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:02:07.953195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.259961+0000","last_change":"2024-08-30T16:05:39.779631+0000","last_active":"2024-08-30T16:05:57.259961+0000","last_peered":"2024-08-30T16:05:57.259961+0000","last_clean":"2024-08-30T16:05:57.259961+0000","last_became_active":"2024-08-30T16:05:39.773479+0000","last_became_peered":"2024-08-30T16:05:39.773479+0000","last_unstale":"2024-08-30T16:05:57.259961+0000","last_undegraded":"2024-08-30T16:05:57.259961+0000","last_fullsized":"2024-08-30T16:05:57.259961+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:29:29.455454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.255363+0000","last_change":"2024-08-30T16:05:40.031358+0000","last_active":"2024-08-30T16:05:57.255363+0000","last_peered":"2024-08-30T16:05:57.255363+0000","last_clean":"2024-08-30T16:05:57.255363+0000","last_became_active":"2024-08-30T16:05:39.776453+0000","last_became_peered":"2024-08-30T16:05:39.776453+0000","last_unstale":"2024-08-30T16:05:57.255363+0000","last_undegraded":"2024-08-30T16:05:57.255363+0000","last_fullsized":"2024-08-30T16:05:57.255363+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:06:14.092608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.462397+0000","last_change":"2024-08-30T16:05:38.365381+0000","last_active":"2024-08-30T16:05:57.462397+0000","last_peered":"2024-08-30T16:05:57.462397+0000","last_clean":"2024-08-30T16:05:57.462397+0000","last_became_active":"2024-08-30T16:05:38.358584+0000","last_became_peered":"2024-08-30T16:05:38.358584+0000","last_unstale":"2024-08-30T16:05:57.462397+0000","last_undegraded":"2024-08-30T16:05:57.462397+0000","last_fullsized":"2024-08-30T16:05:57.462397+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:28:34.896593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":141,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.257472+0000","last_change":"2024-08-30T16:05:38.364964+0000","last_active":"2024-08-30T16:05:57.257472+0000","last_peered":"2024-08-30T16:05:57.257472+0000","last_clean":"2024-08-30T16:05:57.257472+0000","last_became_active":"2024-08-30T16:05:38.358598+0000","last_became_peered":"2024-08-30T16:05:38.358598+0000","last_unstale":"2024-08-30T16:05:57.257472+0000","last_undegraded":"2024-08-30T16:05:57.257472+0000","last_fullsized":"2024-08-30T16:05:57.257472+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T03:36:43.310274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":61,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.904323+0000","last_change":"2024-08-30T16:04:08.039936+0000","last_active":"2024-08-30T16:05:58.904323+0000","last_peered":"2024-08-30T16:05:58.904323+0000","last_clean":"2024-08-30T16:05:58.904323+0000","last_became_active":"2024-08-30T16:04:07.760364+0000","last_became_peered":"2024-08-30T16:04:07.760364+0000","last_unstale":"2024-08-30T16:05:58.904323+0000","last_undegraded":"2024-08-30T16:05:58.904323+0000","last_fullsized":"2024-08-30T16:05:58.904323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:13:37.016725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":160,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.024190+0000","last_change":"2024-08-30T16:05:38.553500+0000","last_active":"2024-08-30T16:05:58.024190+0000","last_peered":"2024-08-30T16:05:58.024190+0000","last_clean":"2024-08-30T16:05:58.024190+0000","last_became_active":"2024-08-30T16:05:38.360837+0000","last_became_peered":"2024-08-30T16:05:38.360837+0000","last_unstale":"2024-08-30T16:05:58.024190+0000","last_undegraded":"2024-08-30T16:05:58.024190+0000","last_fullsized":"2024-08-30T16:05:58.024190+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:15:13.872133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1591360,"data_compressed":3116,"data_compressed_allocated":98304,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280603,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27732,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344108,"statfs":{"total":96636764160,"available":96608366592,"internally_reserved":0,"allocated":724992,"data_stored":658493,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.896000000000001,"5min":11.542,"15min":13.461},"min":{"1min":5.5519999999999996,"5min":4.8049999999999997,"15min":4.8049999999999997},"max":{"1min":25.129999999999999,"5min":51.027000000000001,"15min":57.177999999999997},"last":8.8010000000000002},{"interface":"front","average":{"1min":9.5020000000000007,"5min":11.342000000000001,"15min":12.673999999999999},"min":{"1min":4.5759999999999996,"5min":4.5759999999999996,"15min":4.5759999999999996},"max":{"1min":16.521000000000001,"5min":54.682000000000002,"15min":58.024999999999999},"last":10.916}]},{"osd":2,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.157,"5min":10.984999999999999,"15min":10.614000000000001},"min":{"1min":6.0880000000000001,"5min":5.5780000000000003,"15min":5.5780000000000003},"max":{"1min":15.81,"5min":73.399000000000001,"15min":73.399000000000001},"last":6.1630000000000003},{"interface":"front","average":{"1min":13.478,"5min":14.470000000000001,"15min":14.875},"min":{"1min":6.5640000000000001,"5min":6.0049999999999999,"15min":6.0049999999999999},"max":{"1min":30.704999999999998,"5min":75.120000000000005,"15min":75.120000000000005},"last":13.897}]},{"osd":3,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":13.753,"5min":15.361000000000001,"15min":16.896000000000001},"min":{"1min":9.1199999999999992,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":29.506,"5min":66.869,"15min":66.869},"last":10.417999999999999},{"interface":"front","average":{"1min":10.523999999999999,"5min":13.089,"15min":14.315},"min":{"1min":5.8959999999999999,"5min":5.8959999999999999,"15min":5.8959999999999999},"max":{"1min":16.196000000000002,"5min":104.38200000000001,"15min":104.38200000000001},"last":11.458}]},{"osd":4,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":11.037000000000001,"5min":14.484999999999999,"15min":14.68},"min":{"1min":7.5590000000000002,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":18.202999999999999,"5min":66.340999999999994,"15min":66.340999999999994},"last":13.236000000000001},{"interface":"front","average":{"1min":9.5540000000000003,"5min":12.513999999999999,"15min":12.881},"min":{"1min":6.774,"5min":6.774,"15min":6.774},"max":{"1min":14.335000000000001,"5min":29.818999999999999,"15min":29.818999999999999},"last":8.0860000000000003}]},{"osd":5,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":13.954000000000001,"5min":15.705,"15min":16.510999999999999},"min":{"1min":7.21,"5min":7.21,"15min":7.21},"max":{"1min":31.545000000000002,"5min":55.784999999999997,"15min":55.784999999999997},"last":16.536999999999999},{"interface":"front","average":{"1min":11.759,"5min":17.265999999999998,"15min":18.946000000000002},"min":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"max":{"1min":16.829999999999998,"5min":68.004999999999995,"15min":68.004999999999995},"last":9.3130000000000006}]},{"osd":6,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":15.708,"5min":16.609000000000002,"15min":17.382000000000001},"min":{"1min":10.180999999999999,"5min":9.1929999999999996,"15min":9.1929999999999996},"max":{"1min":27.353999999999999,"5min":58.201999999999998,"15min":58.201999999999998},"last":17.385000000000002},{"interface":"front","average":{"1min":16.324999999999999,"5min":16.748000000000001,"15min":17.001000000000001},"min":{"1min":9.6980000000000004,"5min":8.8390000000000004,"15min":8.8390000000000004},"max":{"1min":26.257999999999999,"5min":37.253,"15min":37.253},"last":12.034000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":16.931000000000001,"5min":17.739999999999998,"15min":19.425999999999998},"min":{"1min":10.352,"5min":8.2050000000000001,"15min":8.2050000000000001},"max":{"1min":34.345999999999997,"5min":68.950000000000003,"15min":68.950000000000003},"last":18.129000000000001},{"interface":"front","average":{"1min":13.039999999999999,"5min":16.266999999999999,"15min":18.443999999999999},"min":{"1min":9.4529999999999994,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":17.625,"5min":36.186,"15min":59.079000000000001},"last":14.454000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313306,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":7.9219999999999997,"5min":8.6739999999999995,"15min":8.8670000000000009},"min":{"1min":4.3789999999999996,"5min":4.3789999999999996,"15min":4.3789999999999996},"max":{"1min":20.73,"5min":25.759,"15min":30.600999999999999},"last":14.135999999999999},{"interface":"front","average":{"1min":7.1589999999999998,"5min":8.8819999999999997,"15min":9.0419999999999998},"min":{"1min":4.8460000000000001,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":12.57,"5min":28.077000000000002,"15min":28.077000000000002},"last":6.0970000000000004}]},{"osd":1,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":11.804,"5min":12.818,"15min":11.496},"min":{"1min":7.0590000000000002,"5min":5.7009999999999996,"15min":5.7009999999999996},"max":{"1min":18.989999999999998,"5min":83.581000000000003,"15min":83.581000000000003},"last":14.663},{"interface":"front","average":{"1min":9.2690000000000001,"5min":11.781000000000001,"15min":12.198},"min":{"1min":5.6230000000000002,"5min":5.6230000000000002,"15min":5.6230000000000002},"max":{"1min":19.524999999999999,"5min":81.722999999999999,"15min":81.722999999999999},"last":7.5270000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":14.83,"5min":13.890000000000001,"15min":13.689},"min":{"1min":8.5039999999999996,"5min":5.4850000000000003,"15min":5.4850000000000003},"max":{"1min":25.885999999999999,"5min":42.506,"15min":42.506},"last":18.111999999999998},{"interface":"front","average":{"1min":13.273999999999999,"5min":13.69,"15min":16.242000000000001},"min":{"1min":6.8029999999999999,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":24.309000000000001,"5min":43.484999999999999,"15min":43.484999999999999},"last":15.276}]},{"osd":3,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":15.515000000000001,"5min":15.986000000000001,"15min":16.391999999999999},"min":{"1min":7.9299999999999997,"5min":7.9299999999999997,"15min":7.9299999999999997},"max":{"1min":54.076999999999998,"5min":65.739999999999995,"15min":65.739999999999995},"last":8.4290000000000003},{"interface":"front","average":{"1min":12.406000000000001,"5min":13.285,"15min":12.871},"min":{"1min":5.6070000000000002,"5min":5.6070000000000002,"15min":5.6070000000000002},"max":{"1min":55.978000000000002,"5min":64.980000000000004,"15min":64.980000000000004},"last":17.135000000000002}]},{"osd":4,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":13.824,"15min":12.827},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.4880000000000004},"max":{"1min":23.213999999999999,"5min":41.631,"15min":41.631},"last":9.6419999999999995},{"interface":"front","average":{"1min":14.590999999999999,"5min":15.433,"15min":17.003},"min":{"1min":9.2789999999999999,"5min":8.8729999999999993,"15min":8.8729999999999993},"max":{"1min":25.149999999999999,"5min":32.526000000000003,"15min":37.393000000000001},"last":12.411}]},{"osd":5,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.68,"5min":12.377000000000001,"15min":11.679},"min":{"1min":6.8259999999999996,"5min":6.2309999999999999,"15min":6.2309999999999999},"max":{"1min":37.372999999999998,"5min":37.372999999999998,"15min":37.372999999999998},"last":16.436},{"interface":"front","average":{"1min":16.558,"5min":15.769,"15min":16.600000000000001},"min":{"1min":9.0429999999999993,"5min":8.2870000000000008,"15min":8.2870000000000008},"max":{"1min":38.484000000000002,"5min":38.618000000000002,"15min":38.618000000000002},"last":10.513}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.326000000000001,"5min":13.771000000000001,"15min":13.208},"min":{"1min":8.0060000000000002,"5min":6.3230000000000004,"15min":6.3230000000000004},"max":{"1min":19.231000000000002,"5min":42.829999999999998,"15min":42.829999999999998},"last":11.218},{"interface":"front","average":{"1min":15.842000000000001,"5min":17.606000000000002,"15min":18.088999999999999},"min":{"1min":11.712,"5min":10.749000000000001,"15min":10.749000000000001},"max":{"1min":22.695,"5min":44.491999999999997,"15min":44.491999999999997},"last":13.292}]}]},{"osd":6,"up_from":105,"seq":450971566091,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":680,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":696320,"data_stored":260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":28176046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.835000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.928000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9849999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.256}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.364000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":696320,"data_stored":260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":9.4120000000000008,"5min":9.5199999999999996,"15min":10.925000000000001},"min":{"1min":5.0830000000000002,"5min":4.3810000000000002,"15min":4.29},"max":{"1min":18.713000000000001,"5min":46.146999999999998,"15min":50.500999999999998},"last":4.4459999999999997},{"interface":"front","average":{"1min":9.7110000000000003,"5min":10.157,"15min":13.363},"min":{"1min":5.9279999999999999,"5min":5.1360000000000001,"15min":5.1360000000000001},"max":{"1min":20.637,"5min":36.720999999999997,"15min":51.965000000000003},"last":6.0869999999999997}]},{"osd":1,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":12.821999999999999,"5min":13.605,"15min":17.742999999999999},"min":{"1min":7.7549999999999999,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":20.864999999999998,"5min":44.862000000000002,"15min":137.56700000000001},"last":8.5299999999999994},{"interface":"front","average":{"1min":10.217000000000001,"5min":12.255000000000001,"15min":15.741},"min":{"1min":6.7839999999999998,"5min":5.0060000000000002,"15min":5.0060000000000002},"max":{"1min":27.032,"5min":47.530999999999999,"15min":136.321},"last":10.907999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":11.897,"5min":12.744999999999999,"15min":15.519},"min":{"1min":5.9109999999999996,"5min":5.9109999999999996,"15min":5.9109999999999996},"max":{"1min":26.527000000000001,"5min":43.869999999999997,"15min":73.831000000000003},"last":9.1300000000000008},{"interface":"front","average":{"1min":11.170999999999999,"5min":11.682,"15min":14.923},"min":{"1min":6.984,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":16.887,"5min":26.074999999999999,"15min":71.831999999999994},"last":11.462}]},{"osd":4,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":14.353,"15min":26.016999999999999},"min":{"1min":7.2350000000000003,"5min":6.9199999999999999,"15min":6.9199999999999999},"max":{"1min":25.645,"5min":43.235999999999997,"15min":342.34500000000003},"last":12.619},{"interface":"front","average":{"1min":12.157,"5min":14.849,"15min":19.047000000000001},"min":{"1min":7.1050000000000004,"5min":6.7859999999999996,"15min":6.7859999999999996},"max":{"1min":23.939,"5min":63.633000000000003,"15min":127.857},"last":10.287000000000001}]},{"osd":5,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":10.967000000000001,"5min":14.295,"15min":15.500999999999999},"min":{"1min":7.1200000000000001,"5min":6.5,"15min":6.5},"max":{"1min":16.876000000000001,"5min":58.715000000000003,"15min":58.715000000000003},"last":14.087999999999999},{"interface":"front","average":{"1min":14.762,"5min":16.952000000000002,"15min":18.649000000000001},"min":{"1min":9.1549999999999994,"5min":7.5869999999999997,"15min":7.2370000000000001},"max":{"1min":29.803999999999998,"5min":58.212000000000003,"15min":58.212000000000003},"last":17.582999999999998}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.295000000000002,"5min":23.492000000000001,"15min":30.053999999999998},"min":{"1min":11.096,"5min":8.6630000000000003,"15min":8.6630000000000003},"max":{"1min":25.298999999999999,"5min":388.96100000000001,"15min":388.96100000000001},"last":15.768000000000001},{"interface":"front","average":{"1min":13.961,"5min":24.18,"15min":32.835999999999999},"min":{"1min":9.4079999999999995,"5min":8.5519999999999996,"15min":8.5519999999999996},"max":{"1min":20.439,"5min":448.69200000000001,"15min":448.69200000000001},"last":16.646999999999998}]},{"osd":7,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":17.667000000000002,"5min":18.959,"15min":24.385000000000002},"min":{"1min":11.335000000000001,"5min":7.3049999999999997,"15min":7.3049999999999997},"max":{"1min":39.133000000000003,"5min":91.646000000000001,"15min":177.596},"last":14.606999999999999},{"interface":"front","average":{"1min":15.132999999999999,"5min":18.663,"15min":25.454999999999998},"min":{"1min":6.891,"5min":6.891,"15min":6.891},"max":{"1min":35.683,"5min":101.158,"15min":231.03800000000001},"last":12.003}]}]},{"osd":4,"up_from":20,"seq":85899346009,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":8.0920000000000005,"5min":8.8539999999999992,"15min":11.156000000000001},"min":{"1min":5.0110000000000001,"5min":4.5439999999999996,"15min":4.5439999999999996},"max":{"1min":11.35,"5min":35.052,"15min":35.817999999999998},"last":5.9390000000000001},{"interface":"front","average":{"1min":7.8849999999999998,"5min":9.2420000000000009,"15min":11.247},"min":{"1min":4.6500000000000004,"5min":4.6500000000000004,"15min":4.6500000000000004},"max":{"1min":16.132999999999999,"5min":35.091999999999999,"15min":35.091999999999999},"last":9.4570000000000007}]},{"osd":1,"last update":"Fri Aug 30 16:05:29 2024","interfaces":[{"interface":"back","average":{"1min":12.667,"5min":12.869,"15min":11.863},"min":{"1min":7.1399999999999997,"5min":5.6879999999999997,"15min":4.7000000000000002},"max":{"1min":24.204000000000001,"5min":34.347999999999999,"15min":34.347999999999999},"last":11.579000000000001},{"interface":"front","average":{"1min":10.877000000000001,"5min":11.917,"15min":12.324},"min":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"max":{"1min":26.788,"5min":35.481000000000002,"15min":35.481000000000002},"last":7.4669999999999996}]},{"osd":2,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":9.6660000000000004,"5min":9.5470000000000006,"15min":9.9280000000000008},"min":{"1min":5.4930000000000003,"5min":5.4279999999999999,"15min":5.4279999999999999},"max":{"1min":30.666,"5min":30.666,"15min":30.666},"last":12.994},{"interface":"front","average":{"1min":11.707000000000001,"5min":12.987,"15min":14.148999999999999},"min":{"1min":7.0629999999999997,"5min":7.0629999999999997,"15min":7.0629999999999997},"max":{"1min":21.355,"5min":29.079999999999998,"15min":30.629000000000001},"last":14.941000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":11.263,"5min":11.942,"15min":12.234999999999999},"min":{"1min":7.476,"5min":5.8010000000000002,"15min":5.8010000000000002},"max":{"1min":28.315999999999999,"5min":29.715,"15min":29.715},"last":15.586},{"interface":"front","average":{"1min":13.385,"5min":14.148999999999999,"15min":15.195},"min":{"1min":7.8220000000000001,"5min":7.5830000000000002,"15min":7.5830000000000002},"max":{"1min":27.321999999999999,"5min":27.321999999999999,"15min":31.73},"last":9.1039999999999992}]},{"osd":5,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":14.595000000000001,"5min":13.505000000000001,"15min":11.943},"min":{"1min":9.1199999999999992,"5min":6.8639999999999999,"15min":6.5010000000000003},"max":{"1min":31.565000000000001,"5min":42.143000000000001,"15min":42.143000000000001},"last":10.592000000000001},{"interface":"front","average":{"1min":12.859,"5min":13.661,"15min":14.194000000000001},"min":{"1min":6.5579999999999998,"5min":6.5579999999999998,"15min":6.5579999999999998},"max":{"1min":29.210999999999999,"5min":43.917000000000002,"15min":43.917000000000002},"last":13.821}]},{"osd":6,"last update":"Fri Aug 30 16:03:23 2024","interfaces":[{"interface":"back","average":{"1min":13.481,"5min":14.797000000000001,"15min":15.696999999999999},"min":{"1min":8.0359999999999996,"5min":6.5289999999999999,"15min":6.5289999999999999},"max":{"1min":20.704000000000001,"5min":37.215000000000003,"15min":37.215000000000003},"last":6.8600000000000003},{"interface":"front","average":{"1min":16.030999999999999,"5min":16.199000000000002,"15min":17.021999999999998},"min":{"1min":9.8300000000000001,"5min":8.7870000000000008,"15min":8.7870000000000008},"max":{"1min":24.986999999999998,"5min":37.753999999999998,"15min":37.753999999999998},"last":11.07}]},{"osd":7,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":14.657,"15min":15.507999999999999},"min":{"1min":7.9950000000000001,"5min":7.7130000000000001,"15min":7.7130000000000001},"max":{"1min":21.992999999999999,"5min":33.314,"15min":34.551000000000002},"last":10.106999999999999},{"interface":"front","average":{"1min":12.443,"5min":15.465,"15min":16.873999999999999},"min":{"1min":6.375,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":18.116,"5min":35.701999999999998,"15min":35.701999999999998},"last":14.300000000000001}]}]},{"osd":2,"up_from":16,"seq":68719476823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:17 2024","interfaces":[{"interface":"back","average":{"1min":10.505000000000001,"5min":9.4960000000000004,"15min":9.6129999999999995},"min":{"1min":4.1200000000000001,"5min":4.1200000000000001,"15min":3.6709999999999998},"max":{"1min":25.526,"5min":29.303000000000001,"15min":30.366},"last":16.681999999999999},{"interface":"front","average":{"1min":11.006,"5min":9.6720000000000006,"15min":10.569000000000001},"min":{"1min":5.4009999999999998,"5min":4.5839999999999996,"15min":3.4900000000000002},"max":{"1min":25.300000000000001,"5min":27.916,"15min":45.604999999999997},"last":36.712000000000003}]},{"osd":1,"last update":"Fri Aug 30 16:05:33 2024","interfaces":[{"interface":"back","average":{"1min":12.826000000000001,"5min":13.023999999999999,"15min":13.119999999999999},"min":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":41.786999999999999,"5min":41.786999999999999,"15min":46.460000000000001},"last":32.917999999999999},{"interface":"front","average":{"1min":15.141,"5min":11.379,"15min":12.558999999999999},"min":{"1min":6.3650000000000002,"5min":5.944,"15min":5.944},"max":{"1min":67.332999999999998,"5min":67.332999999999998,"15min":67.332999999999998},"last":24.684000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":13.191000000000001,"5min":13.573,"15min":16.579000000000001},"min":{"1min":5.7850000000000001,"5min":5.5179999999999998,"15min":5.5179999999999998},"max":{"1min":31.547999999999998,"5min":71.475999999999999,"15min":71.475999999999999},"last":27.98},{"interface":"front","average":{"1min":14.112,"5min":13.813000000000001,"15min":18.425000000000001},"min":{"1min":6.218,"5min":6.218,"15min":6.218},"max":{"1min":40.046999999999997,"5min":40.046999999999997,"15min":50.564},"last":17.902999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":15.548999999999999,"5min":14.409000000000001,"15min":17.457000000000001},"min":{"1min":8.4000000000000004,"5min":7.0970000000000004,"15min":7.0970000000000004},"max":{"1min":28.899000000000001,"5min":33.892000000000003,"15min":48.673999999999999},"last":19.010999999999999},{"interface":"front","average":{"1min":13.239000000000001,"5min":14.494,"15min":15.609999999999999},"min":{"1min":7.1879999999999997,"5min":6.7640000000000002,"15min":6.7640000000000002},"max":{"1min":27.448,"5min":67.054000000000002,"15min":67.054000000000002},"last":22.93}]},{"osd":5,"last update":"Fri Aug 30 16:05:17 2024","interfaces":[{"interface":"back","average":{"1min":13.17,"5min":15.824,"15min":16.684999999999999},"min":{"1min":7.976,"5min":7.0259999999999998,"15min":4.7199999999999998},"max":{"1min":22.773,"5min":69.411000000000001,"15min":69.411000000000001},"last":34.484999999999999},{"interface":"front","average":{"1min":14.057,"5min":16.693000000000001,"15min":18.975000000000001},"min":{"1min":9.5449999999999999,"5min":8.6910000000000007,"15min":5.7460000000000004},"max":{"1min":24.228999999999999,"5min":56.152000000000001,"15min":56.917999999999999},"last":20.838999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:27 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":16.623000000000001,"15min":18.198},"min":{"1min":10.302,"5min":7.4690000000000003,"15min":7.4690000000000003},"max":{"1min":23.317,"5min":72.75,"15min":72.75},"last":19.917000000000002},{"interface":"front","average":{"1min":19.774000000000001,"5min":18.396999999999998,"15min":20.119},"min":{"1min":9.8879999999999999,"5min":8.1579999999999995,"15min":8.1579999999999995},"max":{"1min":70.462999999999994,"5min":74.119,"15min":74.119},"last":26.353000000000002}]},{"osd":7,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":13.157,"5min":17.632000000000001,"15min":20.654},"min":{"1min":8.3160000000000007,"5min":8.2639999999999993,"15min":8.2639999999999993},"max":{"1min":21.460000000000001,"5min":137.33000000000001,"15min":137.33000000000001},"last":29.675000000000001},{"interface":"front","average":{"1min":15.618,"5min":19.195,"15min":19.751000000000001},"min":{"1min":9.7880000000000003,"5min":7.2770000000000001,"15min":7.2770000000000001},"max":{"1min":30.722999999999999,"5min":136.66399999999999,"15min":136.66399999999999},"last":31.329999999999998}]}]},{"osd":0,"up_from":15,"seq":64424509526,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":7.3120000000000003,"5min":10.007,"15min":9.7769999999999992},"min":{"1min":4.8090000000000002,"5min":4.7489999999999997,"15min":4.7489999999999997},"max":{"1min":14.276,"5min":40.735999999999997,"15min":40.735999999999997},"last":9.2279999999999998},{"interface":"front","average":{"1min":10.282,"5min":11.349,"15min":11.352},"min":{"1min":5.0069999999999997,"5min":4.383,"15min":4.383},"max":{"1min":25.077999999999999,"5min":42.387,"15min":42.387},"last":11.018000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":10.721,"5min":11.506,"15min":11.308},"min":{"1min":6.3899999999999997,"5min":5.3689999999999998,"15min":3.3370000000000002},"max":{"1min":24.170999999999999,"5min":46.768999999999998,"15min":46.768999999999998},"last":9.8079999999999998},{"interface":"front","average":{"1min":13.676,"5min":13.199999999999999,"15min":14.022},"min":{"1min":6.8860000000000001,"5min":6.2249999999999996,"15min":5.702},"max":{"1min":31.361000000000001,"5min":43.921999999999997,"15min":43.921999999999997},"last":21.521999999999998}]},{"osd":3,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":11.460000000000001,"5min":13.585000000000001,"15min":21.405000000000001},"min":{"1min":7.702,"5min":6.2839999999999998,"15min":6.2839999999999998},"max":{"1min":29.998999999999999,"5min":38.219000000000001,"15min":200.155},"last":11.916},{"interface":"front","average":{"1min":15.455,"5min":16.989000000000001,"15min":26.460999999999999},"min":{"1min":10.507999999999999,"5min":7.0940000000000003,"15min":7.0940000000000003},"max":{"1min":28.146000000000001,"5min":44.229999999999997,"15min":267.03100000000001},"last":20.286000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.647,"5min":13.015000000000001,"15min":15.007},"min":{"1min":6.891,"5min":6.0970000000000004,"15min":6.0970000000000004},"max":{"1min":14.891,"5min":31.611999999999998,"15min":56.200000000000003},"last":8.7560000000000002},{"interface":"front","average":{"1min":13.050000000000001,"5min":15.571,"15min":15.683999999999999},"min":{"1min":9.3930000000000007,"5min":9.298,"15min":8.609},"max":{"1min":16.626999999999999,"5min":46.875999999999998,"15min":48.534999999999997},"last":14.603999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":14.43,"5min":15.682,"15min":16.082999999999998},"min":{"1min":9.8059999999999992,"5min":5.9199999999999999,"15min":5.6920000000000002},"max":{"1min":40.006,"5min":43.308,"15min":43.308},"last":12.824999999999999},{"interface":"front","average":{"1min":16.951000000000001,"5min":17.693000000000001,"15min":17.853999999999999},"min":{"1min":9.9100000000000001,"5min":9.2149999999999999,"15min":4.75},"max":{"1min":49.575000000000003,"5min":49.575000000000003,"15min":49.575000000000003},"last":17.23}]},{"osd":6,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":13.808999999999999,"5min":14.292,"15min":14.385999999999999},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":39.460999999999999,"5min":59.328000000000003,"15min":59.328000000000003},"last":19.507999999999999},{"interface":"front","average":{"1min":19.225000000000001,"5min":18.331,"15min":18.533999999999999},"min":{"1min":10.781000000000001,"5min":10.005000000000001,"15min":10.005000000000001},"max":{"1min":46.237000000000002,"5min":58.567,"15min":58.567},"last":16.015999999999998}]},{"osd":7,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":18.472000000000001,"5min":19.276,"15min":21.135999999999999},"min":{"1min":11.045,"5min":10.632999999999999,"15min":9.1180000000000003},"max":{"1min":57.804000000000002,"5min":57.804000000000002,"15min":59.515999999999998},"last":22.613},{"interface":"front","average":{"1min":11.272,"5min":13.384,"15min":14.976000000000001},"min":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":7.0590000000000002},"max":{"1min":14.231999999999999,"5min":37.113,"15min":58.807000000000002},"last":17.939}]}]},{"osd":5,"up_from":13,"seq":55834574935,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":658493,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:20 2024","interfaces":[{"interface":"back","average":{"1min":6.2510000000000003,"5min":8.2520000000000007,"15min":12.516999999999999},"min":{"1min":4.0300000000000002,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":13.52,"5min":53.558999999999997,"15min":111.313},"last":9.7219999999999995},{"interface":"front","average":{"1min":8.1809999999999992,"5min":9.4659999999999993,"15min":13.936},"min":{"1min":4.266,"5min":4.266,"15min":4.266},"max":{"1min":14.064,"5min":52.691000000000003,"15min":107.937},"last":7.2809999999999997}]},{"osd":1,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":9.3030000000000008,"5min":10.489000000000001,"15min":12.541},"min":{"1min":5.4539999999999997,"5min":4.657,"15min":4.657},"max":{"1min":16.811,"5min":34.545999999999999,"15min":46.988},"last":6.0279999999999996},{"interface":"front","average":{"1min":10.688000000000001,"5min":12.9,"15min":15.188000000000001},"min":{"1min":7.5179999999999998,"5min":6.4710000000000001,"15min":6.4710000000000001},"max":{"1min":15.455,"5min":33.203000000000003,"15min":47.661000000000001},"last":11.407999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:05:20 2024","interfaces":[{"interface":"back","average":{"1min":13.909000000000001,"5min":15.346,"15min":17.995000000000001},"min":{"1min":8.5630000000000006,"5min":6.9569999999999999,"15min":4.7220000000000004},"max":{"1min":31.667000000000002,"5min":41.549999999999997,"15min":84.317999999999998},"last":10.867000000000001},{"interface":"front","average":{"1min":11.781000000000001,"5min":13.901,"15min":16.207000000000001},"min":{"1min":6.9829999999999997,"5min":6.2770000000000001,"15min":6.0279999999999996},"max":{"1min":29.847999999999999,"5min":43.314999999999998,"15min":99.884},"last":6.7290000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":12.699,"5min":16.114000000000001,"15min":16.768999999999998},"min":{"1min":9.6899999999999995,"5min":6.2149999999999999,"15min":6.2149999999999999},"max":{"1min":18.687000000000001,"5min":117.346,"15min":117.346},"last":10.335000000000001},{"interface":"front","average":{"1min":12.143000000000001,"5min":14.827999999999999,"15min":15.944000000000001},"min":{"1min":6.5270000000000001,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":19.370999999999999,"5min":119.23099999999999,"15min":119.23099999999999},"last":15.724}]},{"osd":4,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":12.651999999999999,"5min":15.768000000000001,"15min":17.32},"min":{"1min":6.5599999999999996,"5min":6.5599999999999996,"15min":6.5599999999999996},"max":{"1min":20.619,"5min":33.582000000000001,"15min":38.095999999999997},"last":12.704000000000001},{"interface":"front","average":{"1min":12.183999999999999,"5min":14.702,"15min":16.603999999999999},"min":{"1min":6.71,"5min":6.71,"15min":6.71},"max":{"1min":20.678999999999998,"5min":31.367000000000001,"15min":40.649000000000001},"last":13.302}]},{"osd":6,"last update":"Fri Aug 30 16:03:26 2024","interfaces":[{"interface":"back","average":{"1min":9.516,"5min":10.686,"15min":11.025},"min":{"1min":6.7809999999999997,"5min":6.367,"15min":6.367},"max":{"1min":15.27,"5min":26.344999999999999,"15min":26.344999999999999},"last":7.8949999999999996},{"interface":"front","average":{"1min":19.596,"5min":17.696999999999999,"15min":17.396000000000001},"min":{"1min":11.433,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":58.052999999999997,"5min":58.052999999999997,"15min":58.052999999999997},"last":14.106}]},{"osd":7,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":13.638,"5min":15.441000000000001,"15min":17.472999999999999},"min":{"1min":7.8490000000000002,"5min":7.8490000000000002,"15min":7.8490000000000002},"max":{"1min":25.056999999999999,"5min":38.180999999999997,"15min":43.604999999999997},"last":15.029999999999999},{"interface":"front","average":{"1min":9.7629999999999999,"5min":11.205,"15min":16.343},"min":{"1min":5.3579999999999997,"5min":5.3579999999999997,"15min":5.3579999999999997},"max":{"1min":22.509,"5min":31.547999999999998,"15min":88.278000000000006},"last":9.2070000000000007}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:06:06.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:06:07.601 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:06:07.602 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:06:07.614 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-08-30T16:06:05.938627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1591398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222156,"kb_used_data":5480,"kb_used_omap":15,"kb_used_meta":216560,"kb_avail":754752564,"statfs":{"total":773094113280,"available":772866625536,"internally_reserved":0,"allocated":5611520,"data_stored":2880942,"data_compressed":1558,"data_compressed_allocated":49152,"data_compressed_original":786432,"omap_allocated":15591,"internal_metadata":221758233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.740013"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.467046+0000","last_change":"2024-08-30T16:04:11.379068+0000","last_active":"2024-08-30T16:05:57.467046+0000","last_peered":"2024-08-30T16:05:57.467046+0000","last_clean":"2024-08-30T16:05:57.467046+0000","last_became_active":"2024-08-30T16:04:11.370524+0000","last_became_peered":"2024-08-30T16:04:11.370524+0000","last_unstale":"2024-08-30T16:05:57.467046+0000","last_undegraded":"2024-08-30T16:05:57.467046+0000","last_fullsized":"2024-08-30T16:05:57.467046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:58:06.441243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.022669+0000","last_change":"2024-08-30T16:04:38.088583+0000","last_active":"2024-08-30T16:05:58.022669+0000","last_peered":"2024-08-30T16:05:58.022669+0000","last_clean":"2024-08-30T16:05:58.022669+0000","last_became_active":"2024-08-30T16:04:38.082821+0000","last_became_peered":"2024-08-30T16:04:38.082821+0000","last_unstale":"2024-08-30T16:05:58.022669+0000","last_undegraded":"2024-08-30T16:05:58.022669+0000","last_fullsized":"2024-08-30T16:05:58.022669+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:40:53.077427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.520451+0000","last_change":"2024-08-30T16:05:38.369602+0000","last_active":"2024-08-30T16:05:57.520451+0000","last_peered":"2024-08-30T16:05:57.520451+0000","last_clean":"2024-08-30T16:05:57.520451+0000","last_became_active":"2024-08-30T16:05:38.361240+0000","last_became_peered":"2024-08-30T16:05:38.361240+0000","last_unstale":"2024-08-30T16:05:57.520451+0000","last_undegraded":"2024-08-30T16:05:57.520451+0000","last_fullsized":"2024-08-30T16:05:57.520451+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:02:07.953195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.259961+0000","last_change":"2024-08-30T16:05:39.779631+0000","last_active":"2024-08-30T16:05:57.259961+0000","last_peered":"2024-08-30T16:05:57.259961+0000","last_clean":"2024-08-30T16:05:57.259961+0000","last_became_active":"2024-08-30T16:05:39.773479+0000","last_became_peered":"2024-08-30T16:05:39.773479+0000","last_unstale":"2024-08-30T16:05:57.259961+0000","last_undegraded":"2024-08-30T16:05:57.259961+0000","last_fullsized":"2024-08-30T16:05:57.259961+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:29:29.455454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.255363+0000","last_change":"2024-08-30T16:05:40.031358+0000","last_active":"2024-08-30T16:05:57.255363+0000","last_peered":"2024-08-30T16:05:57.255363+0000","last_clean":"2024-08-30T16:05:57.255363+0000","last_became_active":"2024-08-30T16:05:39.776453+0000","last_became_peered":"2024-08-30T16:05:39.776453+0000","last_unstale":"2024-08-30T16:05:57.255363+0000","last_undegraded":"2024-08-30T16:05:57.255363+0000","last_fullsized":"2024-08-30T16:05:57.255363+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:06:14.092608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.462397+0000","last_change":"2024-08-30T16:05:38.365381+0000","last_active":"2024-08-30T16:05:57.462397+0000","last_peered":"2024-08-30T16:05:57.462397+0000","last_clean":"2024-08-30T16:05:57.462397+0000","last_became_active":"2024-08-30T16:05:38.358584+0000","last_became_peered":"2024-08-30T16:05:38.358584+0000","last_unstale":"2024-08-30T16:05:57.462397+0000","last_undegraded":"2024-08-30T16:05:57.462397+0000","last_fullsized":"2024-08-30T16:05:57.462397+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:28:34.896593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":141,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.257472+0000","last_change":"2024-08-30T16:05:38.364964+0000","last_active":"2024-08-30T16:05:57.257472+0000","last_peered":"2024-08-30T16:05:57.257472+0000","last_clean":"2024-08-30T16:05:57.257472+0000","last_became_active":"2024-08-30T16:05:38.358598+0000","last_became_peered":"2024-08-30T16:05:38.358598+0000","last_unstale":"2024-08-30T16:05:57.257472+0000","last_undegraded":"2024-08-30T16:05:57.257472+0000","last_fullsized":"2024-08-30T16:05:57.257472+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T03:36:43.310274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":61,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.904323+0000","last_change":"2024-08-30T16:04:08.039936+0000","last_active":"2024-08-30T16:05:58.904323+0000","last_peered":"2024-08-30T16:05:58.904323+0000","last_clean":"2024-08-30T16:05:58.904323+0000","last_became_active":"2024-08-30T16:04:07.760364+0000","last_became_peered":"2024-08-30T16:04:07.760364+0000","last_unstale":"2024-08-30T16:05:58.904323+0000","last_undegraded":"2024-08-30T16:05:58.904323+0000","last_fullsized":"2024-08-30T16:05:58.904323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:13:37.016725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":160,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.024190+0000","last_change":"2024-08-30T16:05:38.553500+0000","last_active":"2024-08-30T16:05:58.024190+0000","last_peered":"2024-08-30T16:05:58.024190+0000","last_clean":"2024-08-30T16:05:58.024190+0000","last_became_active":"2024-08-30T16:05:38.360837+0000","last_became_peered":"2024-08-30T16:05:38.360837+0000","last_unstale":"2024-08-30T16:05:58.024190+0000","last_undegraded":"2024-08-30T16:05:58.024190+0000","last_fullsized":"2024-08-30T16:05:58.024190+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:15:13.872133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1591360,"data_compressed":3116,"data_compressed_allocated":98304,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280603,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27732,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344108,"statfs":{"total":96636764160,"available":96608366592,"internally_reserved":0,"allocated":724992,"data_stored":658493,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.896000000000001,"5min":11.542,"15min":13.461},"min":{"1min":5.5519999999999996,"5min":4.8049999999999997,"15min":4.8049999999999997},"max":{"1min":25.129999999999999,"5min":51.027000000000001,"15min":57.177999999999997},"last":8.8010000000000002},{"interface":"front","average":{"1min":9.5020000000000007,"5min":11.342000000000001,"15min":12.673999999999999},"min":{"1min":4.5759999999999996,"5min":4.5759999999999996,"15min":4.5759999999999996},"max":{"1min":16.521000000000001,"5min":54.682000000000002,"15min":58.024999999999999},"last":10.916}]},{"osd":2,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.157,"5min":10.984999999999999,"15min":10.614000000000001},"min":{"1min":6.0880000000000001,"5min":5.5780000000000003,"15min":5.5780000000000003},"max":{"1min":15.81,"5min":73.399000000000001,"15min":73.399000000000001},"last":6.1630000000000003},{"interface":"front","average":{"1min":13.478,"5min":14.470000000000001,"15min":14.875},"min":{"1min":6.5640000000000001,"5min":6.0049999999999999,"15min":6.0049999999999999},"max":{"1min":30.704999999999998,"5min":75.120000000000005,"15min":75.120000000000005},"last":13.897}]},{"osd":3,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":13.753,"5min":15.361000000000001,"15min":16.896000000000001},"min":{"1min":9.1199999999999992,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":29.506,"5min":66.869,"15min":66.869},"last":10.417999999999999},{"interface":"front","average":{"1min":10.523999999999999,"5min":13.089,"15min":14.315},"min":{"1min":5.8959999999999999,"5min":5.8959999999999999,"15min":5.8959999999999999},"max":{"1min":16.196000000000002,"5min":104.38200000000001,"15min":104.38200000000001},"last":11.458}]},{"osd":4,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":11.037000000000001,"5min":14.484999999999999,"15min":14.68},"min":{"1min":7.5590000000000002,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":18.202999999999999,"5min":66.340999999999994,"15min":66.340999999999994},"last":13.236000000000001},{"interface":"front","average":{"1min":9.5540000000000003,"5min":12.513999999999999,"15min":12.881},"min":{"1min":6.774,"5min":6.774,"15min":6.774},"max":{"1min":14.335000000000001,"5min":29.818999999999999,"15min":29.818999999999999},"last":8.0860000000000003}]},{"osd":5,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":13.954000000000001,"5min":15.705,"15min":16.510999999999999},"min":{"1min":7.21,"5min":7.21,"15min":7.21},"max":{"1min":31.545000000000002,"5min":55.784999999999997,"15min":55.784999999999997},"last":16.536999999999999},{"interface":"front","average":{"1min":11.759,"5min":17.265999999999998,"15min":18.946000000000002},"min":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"max":{"1min":16.829999999999998,"5min":68.004999999999995,"15min":68.004999999999995},"last":9.3130000000000006}]},{"osd":6,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":15.708,"5min":16.609000000000002,"15min":17.382000000000001},"min":{"1min":10.180999999999999,"5min":9.1929999999999996,"15min":9.1929999999999996},"max":{"1min":27.353999999999999,"5min":58.201999999999998,"15min":58.201999999999998},"last":17.385000000000002},{"interface":"front","average":{"1min":16.324999999999999,"5min":16.748000000000001,"15min":17.001000000000001},"min":{"1min":9.6980000000000004,"5min":8.8390000000000004,"15min":8.8390000000000004},"max":{"1min":26.257999999999999,"5min":37.253,"15min":37.253},"last":12.034000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":16.931000000000001,"5min":17.739999999999998,"15min":19.425999999999998},"min":{"1min":10.352,"5min":8.2050000000000001,"15min":8.2050000000000001},"max":{"1min":34.345999999999997,"5min":68.950000000000003,"15min":68.950000000000003},"last":18.129000000000001},{"interface":"front","average":{"1min":13.039999999999999,"5min":16.266999999999999,"15min":18.443999999999999},"min":{"1min":9.4529999999999994,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":17.625,"5min":36.186,"15min":59.079000000000001},"last":14.454000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313306,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":7.9219999999999997,"5min":8.6739999999999995,"15min":8.8670000000000009},"min":{"1min":4.3789999999999996,"5min":4.3789999999999996,"15min":4.3789999999999996},"max":{"1min":20.73,"5min":25.759,"15min":30.600999999999999},"last":14.135999999999999},{"interface":"front","average":{"1min":7.1589999999999998,"5min":8.8819999999999997,"15min":9.0419999999999998},"min":{"1min":4.8460000000000001,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":12.57,"5min":28.077000000000002,"15min":28.077000000000002},"last":6.0970000000000004}]},{"osd":1,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":11.804,"5min":12.818,"15min":11.496},"min":{"1min":7.0590000000000002,"5min":5.7009999999999996,"15min":5.7009999999999996},"max":{"1min":18.989999999999998,"5min":83.581000000000003,"15min":83.581000000000003},"last":14.663},{"interface":"front","average":{"1min":9.2690000000000001,"5min":11.781000000000001,"15min":12.198},"min":{"1min":5.6230000000000002,"5min":5.6230000000000002,"15min":5.6230000000000002},"max":{"1min":19.524999999999999,"5min":81.722999999999999,"15min":81.722999999999999},"last":7.5270000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":14.83,"5min":13.890000000000001,"15min":13.689},"min":{"1min":8.5039999999999996,"5min":5.4850000000000003,"15min":5.4850000000000003},"max":{"1min":25.885999999999999,"5min":42.506,"15min":42.506},"last":18.111999999999998},{"interface":"front","average":{"1min":13.273999999999999,"5min":13.69,"15min":16.242000000000001},"min":{"1min":6.8029999999999999,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":24.309000000000001,"5min":43.484999999999999,"15min":43.484999999999999},"last":15.276}]},{"osd":3,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":15.515000000000001,"5min":15.986000000000001,"15min":16.391999999999999},"min":{"1min":7.9299999999999997,"5min":7.9299999999999997,"15min":7.9299999999999997},"max":{"1min":54.076999999999998,"5min":65.739999999999995,"15min":65.739999999999995},"last":8.4290000000000003},{"interface":"front","average":{"1min":12.406000000000001,"5min":13.285,"15min":12.871},"min":{"1min":5.6070000000000002,"5min":5.6070000000000002,"15min":5.6070000000000002},"max":{"1min":55.978000000000002,"5min":64.980000000000004,"15min":64.980000000000004},"last":17.135000000000002}]},{"osd":4,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":13.824,"15min":12.827},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.4880000000000004},"max":{"1min":23.213999999999999,"5min":41.631,"15min":41.631},"last":9.6419999999999995},{"interface":"front","average":{"1min":14.590999999999999,"5min":15.433,"15min":17.003},"min":{"1min":9.2789999999999999,"5min":8.8729999999999993,"15min":8.8729999999999993},"max":{"1min":25.149999999999999,"5min":32.526000000000003,"15min":37.393000000000001},"last":12.411}]},{"osd":5,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.68,"5min":12.377000000000001,"15min":11.679},"min":{"1min":6.8259999999999996,"5min":6.2309999999999999,"15min":6.2309999999999999},"max":{"1min":37.372999999999998,"5min":37.372999999999998,"15min":37.372999999999998},"last":16.436},{"interface":"front","average":{"1min":16.558,"5min":15.769,"15min":16.600000000000001},"min":{"1min":9.0429999999999993,"5min":8.2870000000000008,"15min":8.2870000000000008},"max":{"1min":38.484000000000002,"5min":38.618000000000002,"15min":38.618000000000002},"last":10.513}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.326000000000001,"5min":13.771000000000001,"15min":13.208},"min":{"1min":8.0060000000000002,"5min":6.3230000000000004,"15min":6.3230000000000004},"max":{"1min":19.231000000000002,"5min":42.829999999999998,"15min":42.829999999999998},"last":11.218},{"interface":"front","average":{"1min":15.842000000000001,"5min":17.606000000000002,"15min":18.088999999999999},"min":{"1min":11.712,"5min":10.749000000000001,"15min":10.749000000000001},"max":{"1min":22.695,"5min":44.491999999999997,"15min":44.491999999999997},"last":13.292}]}]},{"osd":6,"up_from":105,"seq":450971566091,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":680,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":696320,"data_stored":260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":28176046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.835000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.928000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9849999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.256}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.364000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":696320,"data_stored":260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":9.4120000000000008,"5min":9.5199999999999996,"15min":10.925000000000001},"min":{"1min":5.0830000000000002,"5min":4.3810000000000002,"15min":4.29},"max":{"1min":18.713000000000001,"5min":46.146999999999998,"15min":50.500999999999998},"last":4.4459999999999997},{"interface":"front","average":{"1min":9.7110000000000003,"5min":10.157,"15min":13.363},"min":{"1min":5.9279999999999999,"5min":5.1360000000000001,"15min":5.1360000000000001},"max":{"1min":20.637,"5min":36.720999999999997,"15min":51.965000000000003},"last":6.0869999999999997}]},{"osd":1,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":12.821999999999999,"5min":13.605,"15min":17.742999999999999},"min":{"1min":7.7549999999999999,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":20.864999999999998,"5min":44.862000000000002,"15min":137.56700000000001},"last":8.5299999999999994},{"interface":"front","average":{"1min":10.217000000000001,"5min":12.255000000000001,"15min":15.741},"min":{"1min":6.7839999999999998,"5min":5.0060000000000002,"15min":5.0060000000000002},"max":{"1min":27.032,"5min":47.530999999999999,"15min":136.321},"last":10.907999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":11.897,"5min":12.744999999999999,"15min":15.519},"min":{"1min":5.9109999999999996,"5min":5.9109999999999996,"15min":5.9109999999999996},"max":{"1min":26.527000000000001,"5min":43.869999999999997,"15min":73.831000000000003},"last":9.1300000000000008},{"interface":"front","average":{"1min":11.170999999999999,"5min":11.682,"15min":14.923},"min":{"1min":6.984,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":16.887,"5min":26.074999999999999,"15min":71.831999999999994},"last":11.462}]},{"osd":4,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":14.353,"15min":26.016999999999999},"min":{"1min":7.2350000000000003,"5min":6.9199999999999999,"15min":6.9199999999999999},"max":{"1min":25.645,"5min":43.235999999999997,"15min":342.34500000000003},"last":12.619},{"interface":"front","average":{"1min":12.157,"5min":14.849,"15min":19.047000000000001},"min":{"1min":7.1050000000000004,"5min":6.7859999999999996,"15min":6.7859999999999996},"max":{"1min":23.939,"5min":63.633000000000003,"15min":127.857},"last":10.287000000000001}]},{"osd":5,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":10.967000000000001,"5min":14.295,"15min":15.500999999999999},"min":{"1min":7.1200000000000001,"5min":6.5,"15min":6.5},"max":{"1min":16.876000000000001,"5min":58.715000000000003,"15min":58.715000000000003},"last":14.087999999999999},{"interface":"front","average":{"1min":14.762,"5min":16.952000000000002,"15min":18.649000000000001},"min":{"1min":9.1549999999999994,"5min":7.5869999999999997,"15min":7.2370000000000001},"max":{"1min":29.803999999999998,"5min":58.212000000000003,"15min":58.212000000000003},"last":17.582999999999998}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.295000000000002,"5min":23.492000000000001,"15min":30.053999999999998},"min":{"1min":11.096,"5min":8.6630000000000003,"15min":8.6630000000000003},"max":{"1min":25.298999999999999,"5min":388.96100000000001,"15min":388.96100000000001},"last":15.768000000000001},{"interface":"front","average":{"1min":13.961,"5min":24.18,"15min":32.835999999999999},"min":{"1min":9.4079999999999995,"5min":8.5519999999999996,"15min":8.5519999999999996},"max":{"1min":20.439,"5min":448.69200000000001,"15min":448.69200000000001},"last":16.646999999999998}]},{"osd":7,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":17.667000000000002,"5min":18.959,"15min":24.385000000000002},"min":{"1min":11.335000000000001,"5min":7.3049999999999997,"15min":7.3049999999999997},"max":{"1min":39.133000000000003,"5min":91.646000000000001,"15min":177.596},"last":14.606999999999999},{"interface":"front","average":{"1min":15.132999999999999,"5min":18.663,"15min":25.454999999999998},"min":{"1min":6.891,"5min":6.891,"15min":6.891},"max":{"1min":35.683,"5min":101.158,"15min":231.03800000000001},"last":12.003}]}]},{"osd":4,"up_from":20,"seq":85899346009,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":8.0920000000000005,"5min":8.8539999999999992,"15min":11.156000000000001},"min":{"1min":5.0110000000000001,"5min":4.5439999999999996,"15min":4.5439999999999996},"max":{"1min":11.35,"5min":35.052,"15min":35.817999999999998},"last":5.9390000000000001},{"interface":"front","average":{"1min":7.8849999999999998,"5min":9.2420000000000009,"15min":11.247},"min":{"1min":4.6500000000000004,"5min":4.6500000000000004,"15min":4.6500000000000004},"max":{"1min":16.132999999999999,"5min":35.091999999999999,"15min":35.091999999999999},"last":9.4570000000000007}]},{"osd":1,"last update":"Fri Aug 30 16:05:29 2024","interfaces":[{"interface":"back","average":{"1min":12.667,"5min":12.869,"15min":11.863},"min":{"1min":7.1399999999999997,"5min":5.6879999999999997,"15min":4.7000000000000002},"max":{"1min":24.204000000000001,"5min":34.347999999999999,"15min":34.347999999999999},"last":11.579000000000001},{"interface":"front","average":{"1min":10.877000000000001,"5min":11.917,"15min":12.324},"min":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"max":{"1min":26.788,"5min":35.481000000000002,"15min":35.481000000000002},"last":7.4669999999999996}]},{"osd":2,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":9.6660000000000004,"5min":9.5470000000000006,"15min":9.9280000000000008},"min":{"1min":5.4930000000000003,"5min":5.4279999999999999,"15min":5.4279999999999999},"max":{"1min":30.666,"5min":30.666,"15min":30.666},"last":12.994},{"interface":"front","average":{"1min":11.707000000000001,"5min":12.987,"15min":14.148999999999999},"min":{"1min":7.0629999999999997,"5min":7.0629999999999997,"15min":7.0629999999999997},"max":{"1min":21.355,"5min":29.079999999999998,"15min":30.629000000000001},"last":14.941000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":11.263,"5min":11.942,"15min":12.234999999999999},"min":{"1min":7.476,"5min":5.8010000000000002,"15min":5.8010000000000002},"max":{"1min":28.315999999999999,"5min":29.715,"15min":29.715},"last":15.586},{"interface":"front","average":{"1min":13.385,"5min":14.148999999999999,"15min":15.195},"min":{"1min":7.8220000000000001,"5min":7.5830000000000002,"15min":7.5830000000000002},"max":{"1min":27.321999999999999,"5min":27.321999999999999,"15min":31.73},"last":9.1039999999999992}]},{"osd":5,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":14.595000000000001,"5min":13.505000000000001,"15min":11.943},"min":{"1min":9.1199999999999992,"5min":6.8639999999999999,"15min":6.5010000000000003},"max":{"1min":31.565000000000001,"5min":42.143000000000001,"15min":42.143000000000001},"last":10.592000000000001},{"interface":"front","average":{"1min":12.859,"5min":13.661,"15min":14.194000000000001},"min":{"1min":6.5579999999999998,"5min":6.5579999999999998,"15min":6.5579999999999998},"max":{"1min":29.210999999999999,"5min":43.917000000000002,"15min":43.917000000000002},"last":13.821}]},{"osd":6,"last update":"Fri Aug 30 16:03:23 2024","interfaces":[{"interface":"back","average":{"1min":13.481,"5min":14.797000000000001,"15min":15.696999999999999},"min":{"1min":8.0359999999999996,"5min":6.5289999999999999,"15min":6.5289999999999999},"max":{"1min":20.704000000000001,"5min":37.215000000000003,"15min":37.215000000000003},"last":6.8600000000000003},{"interface":"front","average":{"1min":16.030999999999999,"5min":16.199000000000002,"15min":17.021999999999998},"min":{"1min":9.8300000000000001,"5min":8.7870000000000008,"15min":8.7870000000000008},"max":{"1min":24.986999999999998,"5min":37.753999999999998,"15min":37.753999999999998},"last":11.07}]},{"osd":7,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":14.657,"15min":15.507999999999999},"min":{"1min":7.9950000000000001,"5min":7.7130000000000001,"15min":7.7130000000000001},"max":{"1min":21.992999999999999,"5min":33.314,"15min":34.551000000000002},"last":10.106999999999999},{"interface":"front","average":{"1min":12.443,"5min":15.465,"15min":16.873999999999999},"min":{"1min":6.375,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":18.116,"5min":35.701999999999998,"15min":35.701999999999998},"last":14.300000000000001}]}]},{"osd":2,"up_from":16,"seq":68719476823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:17 2024","interfaces":[{"interface":"back","average":{"1min":10.505000000000001,"5min":9.4960000000000004,"15min":9.6129999999999995},"min":{"1min":4.1200000000000001,"5min":4.1200000000000001,"15min":3.6709999999999998},"max":{"1min":25.526,"5min":29.303000000000001,"15min":30.366},"last":16.681999999999999},{"interface":"front","average":{"1min":11.006,"5min":9.6720000000000006,"15min":10.569000000000001},"min":{"1min":5.4009999999999998,"5min":4.5839999999999996,"15min":3.4900000000000002},"max":{"1min":25.300000000000001,"5min":27.916,"15min":45.604999999999997},"last":36.712000000000003}]},{"osd":1,"last update":"Fri Aug 30 16:05:33 2024","interfaces":[{"interface":"back","average":{"1min":12.826000000000001,"5min":13.023999999999999,"15min":13.119999999999999},"min":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":41.786999999999999,"5min":41.786999999999999,"15min":46.460000000000001},"last":32.917999999999999},{"interface":"front","average":{"1min":15.141,"5min":11.379,"15min":12.558999999999999},"min":{"1min":6.3650000000000002,"5min":5.944,"15min":5.944},"max":{"1min":67.332999999999998,"5min":67.332999999999998,"15min":67.332999999999998},"last":24.684000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":13.191000000000001,"5min":13.573,"15min":16.579000000000001},"min":{"1min":5.7850000000000001,"5min":5.5179999999999998,"15min":5.5179999999999998},"max":{"1min":31.547999999999998,"5min":71.475999999999999,"15min":71.475999999999999},"last":27.98},{"interface":"front","average":{"1min":14.112,"5min":13.813000000000001,"15min":18.425000000000001},"min":{"1min":6.218,"5min":6.218,"15min":6.218},"max":{"1min":40.046999999999997,"5min":40.046999999999997,"15min":50.564},"last":17.902999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":15.548999999999999,"5min":14.409000000000001,"15min":17.457000000000001},"min":{"1min":8.4000000000000004,"5min":7.0970000000000004,"15min":7.0970000000000004},"max":{"1min":28.899000000000001,"5min":33.892000000000003,"15min":48.673999999999999},"last":19.010999999999999},{"interface":"front","average":{"1min":13.239000000000001,"5min":14.494,"15min":15.609999999999999},"min":{"1min":7.1879999999999997,"5min":6.7640000000000002,"15min":6.7640000000000002},"max":{"1min":27.448,"5min":67.054000000000002,"15min":67.054000000000002},"last":22.93}]},{"osd":5,"last update":"Fri Aug 30 16:05:17 2024","interfaces":[{"interface":"back","average":{"1min":13.17,"5min":15.824,"15min":16.684999999999999},"min":{"1min":7.976,"5min":7.0259999999999998,"15min":4.7199999999999998},"max":{"1min":22.773,"5min":69.411000000000001,"15min":69.411000000000001},"last":34.484999999999999},{"interface":"front","average":{"1min":14.057,"5min":16.693000000000001,"15min":18.975000000000001},"min":{"1min":9.5449999999999999,"5min":8.6910000000000007,"15min":5.7460000000000004},"max":{"1min":24.228999999999999,"5min":56.152000000000001,"15min":56.917999999999999},"last":20.838999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:27 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":16.623000000000001,"15min":18.198},"min":{"1min":10.302,"5min":7.4690000000000003,"15min":7.4690000000000003},"max":{"1min":23.317,"5min":72.75,"15min":72.75},"last":19.917000000000002},{"interface":"front","average":{"1min":19.774000000000001,"5min":18.396999999999998,"15min":20.119},"min":{"1min":9.8879999999999999,"5min":8.1579999999999995,"15min":8.1579999999999995},"max":{"1min":70.462999999999994,"5min":74.119,"15min":74.119},"last":26.353000000000002}]},{"osd":7,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":13.157,"5min":17.632000000000001,"15min":20.654},"min":{"1min":8.3160000000000007,"5min":8.2639999999999993,"15min":8.2639999999999993},"max":{"1min":21.460000000000001,"5min":137.33000000000001,"15min":137.33000000000001},"last":29.675000000000001},{"interface":"front","average":{"1min":15.618,"5min":19.195,"15min":19.751000000000001},"min":{"1min":9.7880000000000003,"5min":7.2770000000000001,"15min":7.2770000000000001},"max":{"1min":30.722999999999999,"5min":136.66399999999999,"15min":136.66399999999999},"last":31.329999999999998}]}]},{"osd":0,"up_from":15,"seq":64424509526,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":7.3120000000000003,"5min":10.007,"15min":9.7769999999999992},"min":{"1min":4.8090000000000002,"5min":4.7489999999999997,"15min":4.7489999999999997},"max":{"1min":14.276,"5min":40.735999999999997,"15min":40.735999999999997},"last":9.2279999999999998},{"interface":"front","average":{"1min":10.282,"5min":11.349,"15min":11.352},"min":{"1min":5.0069999999999997,"5min":4.383,"15min":4.383},"max":{"1min":25.077999999999999,"5min":42.387,"15min":42.387},"last":11.018000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":10.721,"5min":11.506,"15min":11.308},"min":{"1min":6.3899999999999997,"5min":5.3689999999999998,"15min":3.3370000000000002},"max":{"1min":24.170999999999999,"5min":46.768999999999998,"15min":46.768999999999998},"last":9.8079999999999998},{"interface":"front","average":{"1min":13.676,"5min":13.199999999999999,"15min":14.022},"min":{"1min":6.8860000000000001,"5min":6.2249999999999996,"15min":5.702},"max":{"1min":31.361000000000001,"5min":43.921999999999997,"15min":43.921999999999997},"last":21.521999999999998}]},{"osd":3,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":11.460000000000001,"5min":13.585000000000001,"15min":21.405000000000001},"min":{"1min":7.702,"5min":6.2839999999999998,"15min":6.2839999999999998},"max":{"1min":29.998999999999999,"5min":38.219000000000001,"15min":200.155},"last":11.916},{"interface":"front","average":{"1min":15.455,"5min":16.989000000000001,"15min":26.460999999999999},"min":{"1min":10.507999999999999,"5min":7.0940000000000003,"15min":7.0940000000000003},"max":{"1min":28.146000000000001,"5min":44.229999999999997,"15min":267.03100000000001},"last":20.286000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.647,"5min":13.015000000000001,"15min":15.007},"min":{"1min":6.891,"5min":6.0970000000000004,"15min":6.0970000000000004},"max":{"1min":14.891,"5min":31.611999999999998,"15min":56.200000000000003},"last":8.7560000000000002},{"interface":"front","average":{"1min":13.050000000000001,"5min":15.571,"15min":15.683999999999999},"min":{"1min":9.3930000000000007,"5min":9.298,"15min":8.609},"max":{"1min":16.626999999999999,"5min":46.875999999999998,"15min":48.534999999999997},"last":14.603999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":14.43,"5min":15.682,"15min":16.082999999999998},"min":{"1min":9.8059999999999992,"5min":5.9199999999999999,"15min":5.6920000000000002},"max":{"1min":40.006,"5min":43.308,"15min":43.308},"last":12.824999999999999},{"interface":"front","average":{"1min":16.951000000000001,"5min":17.693000000000001,"15min":17.853999999999999},"min":{"1min":9.9100000000000001,"5min":9.2149999999999999,"15min":4.75},"max":{"1min":49.575000000000003,"5min":49.575000000000003,"15min":49.575000000000003},"last":17.23}]},{"osd":6,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":13.808999999999999,"5min":14.292,"15min":14.385999999999999},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":39.460999999999999,"5min":59.328000000000003,"15min":59.328000000000003},"last":19.507999999999999},{"interface":"front","average":{"1min":19.225000000000001,"5min":18.331,"15min":18.533999999999999},"min":{"1min":10.781000000000001,"5min":10.005000000000001,"15min":10.005000000000001},"max":{"1min":46.237000000000002,"5min":58.567,"15min":58.567},"last":16.015999999999998}]},{"osd":7,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":18.472000000000001,"5min":19.276,"15min":21.135999999999999},"min":{"1min":11.045,"5min":10.632999999999999,"15min":9.1180000000000003},"max":{"1min":57.804000000000002,"5min":57.804000000000002,"15min":59.515999999999998},"last":22.613},{"interface":"front","average":{"1min":11.272,"5min":13.384,"15min":14.976000000000001},"min":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":7.0590000000000002},"max":{"1min":14.231999999999999,"5min":37.113,"15min":58.807000000000002},"last":17.939}]}]},{"osd":5,"up_from":13,"seq":55834574935,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":658493,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:20 2024","interfaces":[{"interface":"back","average":{"1min":6.2510000000000003,"5min":8.2520000000000007,"15min":12.516999999999999},"min":{"1min":4.0300000000000002,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":13.52,"5min":53.558999999999997,"15min":111.313},"last":9.7219999999999995},{"interface":"front","average":{"1min":8.1809999999999992,"5min":9.4659999999999993,"15min":13.936},"min":{"1min":4.266,"5min":4.266,"15min":4.266},"max":{"1min":14.064,"5min":52.691000000000003,"15min":107.937},"last":7.2809999999999997}]},{"osd":1,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":9.3030000000000008,"5min":10.489000000000001,"15min":12.541},"min":{"1min":5.4539999999999997,"5min":4.657,"15min":4.657},"max":{"1min":16.811,"5min":34.545999999999999,"15min":46.988},"last":6.0279999999999996},{"interface":"front","average":{"1min":10.688000000000001,"5min":12.9,"15min":15.188000000000001},"min":{"1min":7.5179999999999998,"5min":6.4710000000000001,"15min":6.4710000000000001},"max":{"1min":15.455,"5min":33.203000000000003,"15min":47.661000000000001},"last":11.407999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:05:20 2024","interfaces":[{"interface":"back","average":{"1min":13.909000000000001,"5min":15.346,"15min":17.995000000000001},"min":{"1min":8.5630000000000006,"5min":6.9569999999999999,"15min":4.7220000000000004},"max":{"1min":31.667000000000002,"5min":41.549999999999997,"15min":84.317999999999998},"last":10.867000000000001},{"interface":"front","average":{"1min":11.781000000000001,"5min":13.901,"15min":16.207000000000001},"min":{"1min":6.9829999999999997,"5min":6.2770000000000001,"15min":6.0279999999999996},"max":{"1min":29.847999999999999,"5min":43.314999999999998,"15min":99.884},"last":6.7290000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":12.699,"5min":16.114000000000001,"15min":16.768999999999998},"min":{"1min":9.6899999999999995,"5min":6.2149999999999999,"15min":6.2149999999999999},"max":{"1min":18.687000000000001,"5min":117.346,"15min":117.346},"last":10.335000000000001},{"interface":"front","average":{"1min":12.143000000000001,"5min":14.827999999999999,"15min":15.944000000000001},"min":{"1min":6.5270000000000001,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":19.370999999999999,"5min":119.23099999999999,"15min":119.23099999999999},"last":15.724}]},{"osd":4,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":12.651999999999999,"5min":15.768000000000001,"15min":17.32},"min":{"1min":6.5599999999999996,"5min":6.5599999999999996,"15min":6.5599999999999996},"max":{"1min":20.619,"5min":33.582000000000001,"15min":38.095999999999997},"last":12.704000000000001},{"interface":"front","average":{"1min":12.183999999999999,"5min":14.702,"15min":16.603999999999999},"min":{"1min":6.71,"5min":6.71,"15min":6.71},"max":{"1min":20.678999999999998,"5min":31.367000000000001,"15min":40.649000000000001},"last":13.302}]},{"osd":6,"last update":"Fri Aug 30 16:03:26 2024","interfaces":[{"interface":"back","average":{"1min":9.516,"5min":10.686,"15min":11.025},"min":{"1min":6.7809999999999997,"5min":6.367,"15min":6.367},"max":{"1min":15.27,"5min":26.344999999999999,"15min":26.344999999999999},"last":7.8949999999999996},{"interface":"front","average":{"1min":19.596,"5min":17.696999999999999,"15min":17.396000000000001},"min":{"1min":11.433,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":58.052999999999997,"5min":58.052999999999997,"15min":58.052999999999997},"last":14.106}]},{"osd":7,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":13.638,"5min":15.441000000000001,"15min":17.472999999999999},"min":{"1min":7.8490000000000002,"5min":7.8490000000000002,"15min":7.8490000000000002},"max":{"1min":25.056999999999999,"5min":38.180999999999997,"15min":43.604999999999997},"last":15.029999999999999},{"interface":"front","average":{"1min":9.7629999999999999,"5min":11.205,"15min":16.343},"min":{"1min":5.3579999999999997,"5min":5.3579999999999997,"15min":5.3579999999999997},"max":{"1min":22.509,"5min":31.547999999999998,"15min":88.278000000000006},"last":9.2070000000000007}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:06:07.615 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-30T16:06:07.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:06:08.177 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:06:08.177 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:06:08.189 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-08-30T16:06:07.938932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1591398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222156,"kb_used_data":5480,"kb_used_omap":15,"kb_used_meta":216560,"kb_avail":754752564,"statfs":{"total":773094113280,"available":772866625536,"internally_reserved":0,"allocated":5611520,"data_stored":2880942,"data_compressed":1558,"data_compressed_allocated":49152,"data_compressed_original":786432,"omap_allocated":15591,"internal_metadata":221758233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001737"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.467046+0000","last_change":"2024-08-30T16:04:11.379068+0000","last_active":"2024-08-30T16:05:57.467046+0000","last_peered":"2024-08-30T16:05:57.467046+0000","last_clean":"2024-08-30T16:05:57.467046+0000","last_became_active":"2024-08-30T16:04:11.370524+0000","last_became_peered":"2024-08-30T16:04:11.370524+0000","last_unstale":"2024-08-30T16:05:57.467046+0000","last_undegraded":"2024-08-30T16:05:57.467046+0000","last_fullsized":"2024-08-30T16:05:57.467046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:58:06.441243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.022669+0000","last_change":"2024-08-30T16:04:38.088583+0000","last_active":"2024-08-30T16:05:58.022669+0000","last_peered":"2024-08-30T16:05:58.022669+0000","last_clean":"2024-08-30T16:05:58.022669+0000","last_became_active":"2024-08-30T16:04:38.082821+0000","last_became_peered":"2024-08-30T16:04:38.082821+0000","last_unstale":"2024-08-30T16:05:58.022669+0000","last_undegraded":"2024-08-30T16:05:58.022669+0000","last_fullsized":"2024-08-30T16:05:58.022669+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:40:53.077427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.520451+0000","last_change":"2024-08-30T16:05:38.369602+0000","last_active":"2024-08-30T16:05:57.520451+0000","last_peered":"2024-08-30T16:05:57.520451+0000","last_clean":"2024-08-30T16:05:57.520451+0000","last_became_active":"2024-08-30T16:05:38.361240+0000","last_became_peered":"2024-08-30T16:05:38.361240+0000","last_unstale":"2024-08-30T16:05:57.520451+0000","last_undegraded":"2024-08-30T16:05:57.520451+0000","last_fullsized":"2024-08-30T16:05:57.520451+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:02:07.953195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.259961+0000","last_change":"2024-08-30T16:05:39.779631+0000","last_active":"2024-08-30T16:05:57.259961+0000","last_peered":"2024-08-30T16:05:57.259961+0000","last_clean":"2024-08-30T16:05:57.259961+0000","last_became_active":"2024-08-30T16:05:39.773479+0000","last_became_peered":"2024-08-30T16:05:39.773479+0000","last_unstale":"2024-08-30T16:05:57.259961+0000","last_undegraded":"2024-08-30T16:05:57.259961+0000","last_fullsized":"2024-08-30T16:05:57.259961+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:29:29.455454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.255363+0000","last_change":"2024-08-30T16:05:40.031358+0000","last_active":"2024-08-30T16:05:57.255363+0000","last_peered":"2024-08-30T16:05:57.255363+0000","last_clean":"2024-08-30T16:05:57.255363+0000","last_became_active":"2024-08-30T16:05:39.776453+0000","last_became_peered":"2024-08-30T16:05:39.776453+0000","last_unstale":"2024-08-30T16:05:57.255363+0000","last_undegraded":"2024-08-30T16:05:57.255363+0000","last_fullsized":"2024-08-30T16:05:57.255363+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T19:06:14.092608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.462397+0000","last_change":"2024-08-30T16:05:38.365381+0000","last_active":"2024-08-30T16:05:57.462397+0000","last_peered":"2024-08-30T16:05:57.462397+0000","last_clean":"2024-08-30T16:05:57.462397+0000","last_became_active":"2024-08-30T16:05:38.358584+0000","last_became_peered":"2024-08-30T16:05:38.358584+0000","last_unstale":"2024-08-30T16:05:57.462397+0000","last_undegraded":"2024-08-30T16:05:57.462397+0000","last_fullsized":"2024-08-30T16:05:57.462397+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:28:34.896593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":141,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.257472+0000","last_change":"2024-08-30T16:05:38.364964+0000","last_active":"2024-08-30T16:05:57.257472+0000","last_peered":"2024-08-30T16:05:57.257472+0000","last_clean":"2024-08-30T16:05:57.257472+0000","last_became_active":"2024-08-30T16:05:38.358598+0000","last_became_peered":"2024-08-30T16:05:38.358598+0000","last_unstale":"2024-08-30T16:05:57.257472+0000","last_undegraded":"2024-08-30T16:05:57.257472+0000","last_fullsized":"2024-08-30T16:05:57.257472+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T03:36:43.310274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":61,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.904323+0000","last_change":"2024-08-30T16:04:08.039936+0000","last_active":"2024-08-30T16:05:58.904323+0000","last_peered":"2024-08-30T16:05:58.904323+0000","last_clean":"2024-08-30T16:05:58.904323+0000","last_became_active":"2024-08-30T16:04:07.760364+0000","last_became_peered":"2024-08-30T16:04:07.760364+0000","last_unstale":"2024-08-30T16:05:58.904323+0000","last_undegraded":"2024-08-30T16:05:58.904323+0000","last_fullsized":"2024-08-30T16:05:58.904323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:13:37.016725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":160,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.024190+0000","last_change":"2024-08-30T16:05:38.553500+0000","last_active":"2024-08-30T16:05:58.024190+0000","last_peered":"2024-08-30T16:05:58.024190+0000","last_clean":"2024-08-30T16:05:58.024190+0000","last_became_active":"2024-08-30T16:05:38.360837+0000","last_became_peered":"2024-08-30T16:05:38.360837+0000","last_unstale":"2024-08-30T16:05:58.024190+0000","last_undegraded":"2024-08-30T16:05:58.024190+0000","last_fullsized":"2024-08-30T16:05:58.024190+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:15:13.872133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1591360,"data_compressed":3116,"data_compressed_allocated":98304,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280603,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27732,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344108,"statfs":{"total":96636764160,"available":96608366592,"internally_reserved":0,"allocated":724992,"data_stored":658493,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.896000000000001,"5min":11.542,"15min":13.461},"min":{"1min":5.5519999999999996,"5min":4.8049999999999997,"15min":4.8049999999999997},"max":{"1min":25.129999999999999,"5min":51.027000000000001,"15min":57.177999999999997},"last":8.8010000000000002},{"interface":"front","average":{"1min":9.5020000000000007,"5min":11.342000000000001,"15min":12.673999999999999},"min":{"1min":4.5759999999999996,"5min":4.5759999999999996,"15min":4.5759999999999996},"max":{"1min":16.521000000000001,"5min":54.682000000000002,"15min":58.024999999999999},"last":10.916}]},{"osd":2,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.157,"5min":10.984999999999999,"15min":10.614000000000001},"min":{"1min":6.0880000000000001,"5min":5.5780000000000003,"15min":5.5780000000000003},"max":{"1min":15.81,"5min":73.399000000000001,"15min":73.399000000000001},"last":6.1630000000000003},{"interface":"front","average":{"1min":13.478,"5min":14.470000000000001,"15min":14.875},"min":{"1min":6.5640000000000001,"5min":6.0049999999999999,"15min":6.0049999999999999},"max":{"1min":30.704999999999998,"5min":75.120000000000005,"15min":75.120000000000005},"last":13.897}]},{"osd":3,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":13.753,"5min":15.361000000000001,"15min":16.896000000000001},"min":{"1min":9.1199999999999992,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":29.506,"5min":66.869,"15min":66.869},"last":10.417999999999999},{"interface":"front","average":{"1min":10.523999999999999,"5min":13.089,"15min":14.315},"min":{"1min":5.8959999999999999,"5min":5.8959999999999999,"15min":5.8959999999999999},"max":{"1min":16.196000000000002,"5min":104.38200000000001,"15min":104.38200000000001},"last":11.458}]},{"osd":4,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":11.037000000000001,"5min":14.484999999999999,"15min":14.68},"min":{"1min":7.5590000000000002,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":18.202999999999999,"5min":66.340999999999994,"15min":66.340999999999994},"last":13.236000000000001},{"interface":"front","average":{"1min":9.5540000000000003,"5min":12.513999999999999,"15min":12.881},"min":{"1min":6.774,"5min":6.774,"15min":6.774},"max":{"1min":14.335000000000001,"5min":29.818999999999999,"15min":29.818999999999999},"last":8.0860000000000003}]},{"osd":5,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":13.954000000000001,"5min":15.705,"15min":16.510999999999999},"min":{"1min":7.21,"5min":7.21,"15min":7.21},"max":{"1min":31.545000000000002,"5min":55.784999999999997,"15min":55.784999999999997},"last":16.536999999999999},{"interface":"front","average":{"1min":11.759,"5min":17.265999999999998,"15min":18.946000000000002},"min":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"max":{"1min":16.829999999999998,"5min":68.004999999999995,"15min":68.004999999999995},"last":9.3130000000000006}]},{"osd":6,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":15.708,"5min":16.609000000000002,"15min":17.382000000000001},"min":{"1min":10.180999999999999,"5min":9.1929999999999996,"15min":9.1929999999999996},"max":{"1min":27.353999999999999,"5min":58.201999999999998,"15min":58.201999999999998},"last":17.385000000000002},{"interface":"front","average":{"1min":16.324999999999999,"5min":16.748000000000001,"15min":17.001000000000001},"min":{"1min":9.6980000000000004,"5min":8.8390000000000004,"15min":8.8390000000000004},"max":{"1min":26.257999999999999,"5min":37.253,"15min":37.253},"last":12.034000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":16.931000000000001,"5min":17.739999999999998,"15min":19.425999999999998},"min":{"1min":10.352,"5min":8.2050000000000001,"15min":8.2050000000000001},"max":{"1min":34.345999999999997,"5min":68.950000000000003,"15min":68.950000000000003},"last":18.129000000000001},{"interface":"front","average":{"1min":13.039999999999999,"5min":16.266999999999999,"15min":18.443999999999999},"min":{"1min":9.4529999999999994,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":17.625,"5min":36.186,"15min":59.079000000000001},"last":14.454000000000001}]}]},{"osd":7,"up_from":21,"seq":90194313306,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":7.9219999999999997,"5min":8.6739999999999995,"15min":8.8670000000000009},"min":{"1min":4.3789999999999996,"5min":4.3789999999999996,"15min":4.3789999999999996},"max":{"1min":20.73,"5min":25.759,"15min":30.600999999999999},"last":14.135999999999999},{"interface":"front","average":{"1min":7.1589999999999998,"5min":8.8819999999999997,"15min":9.0419999999999998},"min":{"1min":4.8460000000000001,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":12.57,"5min":28.077000000000002,"15min":28.077000000000002},"last":6.0970000000000004}]},{"osd":1,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":11.804,"5min":12.818,"15min":11.496},"min":{"1min":7.0590000000000002,"5min":5.7009999999999996,"15min":5.7009999999999996},"max":{"1min":18.989999999999998,"5min":83.581000000000003,"15min":83.581000000000003},"last":14.663},{"interface":"front","average":{"1min":9.2690000000000001,"5min":11.781000000000001,"15min":12.198},"min":{"1min":5.6230000000000002,"5min":5.6230000000000002,"15min":5.6230000000000002},"max":{"1min":19.524999999999999,"5min":81.722999999999999,"15min":81.722999999999999},"last":7.5270000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":14.83,"5min":13.890000000000001,"15min":13.689},"min":{"1min":8.5039999999999996,"5min":5.4850000000000003,"15min":5.4850000000000003},"max":{"1min":25.885999999999999,"5min":42.506,"15min":42.506},"last":18.111999999999998},{"interface":"front","average":{"1min":13.273999999999999,"5min":13.69,"15min":16.242000000000001},"min":{"1min":6.8029999999999999,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":24.309000000000001,"5min":43.484999999999999,"15min":43.484999999999999},"last":15.276}]},{"osd":3,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":15.515000000000001,"5min":15.986000000000001,"15min":16.391999999999999},"min":{"1min":7.9299999999999997,"5min":7.9299999999999997,"15min":7.9299999999999997},"max":{"1min":54.076999999999998,"5min":65.739999999999995,"15min":65.739999999999995},"last":8.4290000000000003},{"interface":"front","average":{"1min":12.406000000000001,"5min":13.285,"15min":12.871},"min":{"1min":5.6070000000000002,"5min":5.6070000000000002,"15min":5.6070000000000002},"max":{"1min":55.978000000000002,"5min":64.980000000000004,"15min":64.980000000000004},"last":17.135000000000002}]},{"osd":4,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":13.824,"15min":12.827},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.4880000000000004},"max":{"1min":23.213999999999999,"5min":41.631,"15min":41.631},"last":9.6419999999999995},{"interface":"front","average":{"1min":14.590999999999999,"5min":15.433,"15min":17.003},"min":{"1min":9.2789999999999999,"5min":8.8729999999999993,"15min":8.8729999999999993},"max":{"1min":25.149999999999999,"5min":32.526000000000003,"15min":37.393000000000001},"last":12.411}]},{"osd":5,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.68,"5min":12.377000000000001,"15min":11.679},"min":{"1min":6.8259999999999996,"5min":6.2309999999999999,"15min":6.2309999999999999},"max":{"1min":37.372999999999998,"5min":37.372999999999998,"15min":37.372999999999998},"last":16.436},{"interface":"front","average":{"1min":16.558,"5min":15.769,"15min":16.600000000000001},"min":{"1min":9.0429999999999993,"5min":8.2870000000000008,"15min":8.2870000000000008},"max":{"1min":38.484000000000002,"5min":38.618000000000002,"15min":38.618000000000002},"last":10.513}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.326000000000001,"5min":13.771000000000001,"15min":13.208},"min":{"1min":8.0060000000000002,"5min":6.3230000000000004,"15min":6.3230000000000004},"max":{"1min":19.231000000000002,"5min":42.829999999999998,"15min":42.829999999999998},"last":11.218},{"interface":"front","average":{"1min":15.842000000000001,"5min":17.606000000000002,"15min":18.088999999999999},"min":{"1min":11.712,"5min":10.749000000000001,"15min":10.749000000000001},"max":{"1min":22.695,"5min":44.491999999999997,"15min":44.491999999999997},"last":13.292}]}]},{"osd":6,"up_from":105,"seq":450971566091,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":680,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":696320,"data_stored":260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":28176046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.835000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.928000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9849999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.256}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.364000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":696320,"data_stored":260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":9.4120000000000008,"5min":9.5199999999999996,"15min":10.925000000000001},"min":{"1min":5.0830000000000002,"5min":4.3810000000000002,"15min":4.29},"max":{"1min":18.713000000000001,"5min":46.146999999999998,"15min":50.500999999999998},"last":4.4459999999999997},{"interface":"front","average":{"1min":9.7110000000000003,"5min":10.157,"15min":13.363},"min":{"1min":5.9279999999999999,"5min":5.1360000000000001,"15min":5.1360000000000001},"max":{"1min":20.637,"5min":36.720999999999997,"15min":51.965000000000003},"last":6.0869999999999997}]},{"osd":1,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":12.821999999999999,"5min":13.605,"15min":17.742999999999999},"min":{"1min":7.7549999999999999,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":20.864999999999998,"5min":44.862000000000002,"15min":137.56700000000001},"last":8.5299999999999994},{"interface":"front","average":{"1min":10.217000000000001,"5min":12.255000000000001,"15min":15.741},"min":{"1min":6.7839999999999998,"5min":5.0060000000000002,"15min":5.0060000000000002},"max":{"1min":27.032,"5min":47.530999999999999,"15min":136.321},"last":10.907999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":11.897,"5min":12.744999999999999,"15min":15.519},"min":{"1min":5.9109999999999996,"5min":5.9109999999999996,"15min":5.9109999999999996},"max":{"1min":26.527000000000001,"5min":43.869999999999997,"15min":73.831000000000003},"last":9.1300000000000008},{"interface":"front","average":{"1min":11.170999999999999,"5min":11.682,"15min":14.923},"min":{"1min":6.984,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":16.887,"5min":26.074999999999999,"15min":71.831999999999994},"last":11.462}]},{"osd":4,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":14.353,"15min":26.016999999999999},"min":{"1min":7.2350000000000003,"5min":6.9199999999999999,"15min":6.9199999999999999},"max":{"1min":25.645,"5min":43.235999999999997,"15min":342.34500000000003},"last":12.619},{"interface":"front","average":{"1min":12.157,"5min":14.849,"15min":19.047000000000001},"min":{"1min":7.1050000000000004,"5min":6.7859999999999996,"15min":6.7859999999999996},"max":{"1min":23.939,"5min":63.633000000000003,"15min":127.857},"last":10.287000000000001}]},{"osd":5,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":10.967000000000001,"5min":14.295,"15min":15.500999999999999},"min":{"1min":7.1200000000000001,"5min":6.5,"15min":6.5},"max":{"1min":16.876000000000001,"5min":58.715000000000003,"15min":58.715000000000003},"last":14.087999999999999},{"interface":"front","average":{"1min":14.762,"5min":16.952000000000002,"15min":18.649000000000001},"min":{"1min":9.1549999999999994,"5min":7.5869999999999997,"15min":7.2370000000000001},"max":{"1min":29.803999999999998,"5min":58.212000000000003,"15min":58.212000000000003},"last":17.582999999999998}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.295000000000002,"5min":23.492000000000001,"15min":30.053999999999998},"min":{"1min":11.096,"5min":8.6630000000000003,"15min":8.6630000000000003},"max":{"1min":25.298999999999999,"5min":388.96100000000001,"15min":388.96100000000001},"last":15.768000000000001},{"interface":"front","average":{"1min":13.961,"5min":24.18,"15min":32.835999999999999},"min":{"1min":9.4079999999999995,"5min":8.5519999999999996,"15min":8.5519999999999996},"max":{"1min":20.439,"5min":448.69200000000001,"15min":448.69200000000001},"last":16.646999999999998}]},{"osd":7,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":17.667000000000002,"5min":18.959,"15min":24.385000000000002},"min":{"1min":11.335000000000001,"5min":7.3049999999999997,"15min":7.3049999999999997},"max":{"1min":39.133000000000003,"5min":91.646000000000001,"15min":177.596},"last":14.606999999999999},{"interface":"front","average":{"1min":15.132999999999999,"5min":18.663,"15min":25.454999999999998},"min":{"1min":6.891,"5min":6.891,"15min":6.891},"max":{"1min":35.683,"5min":101.158,"15min":231.03800000000001},"last":12.003}]}]},{"osd":4,"up_from":20,"seq":85899346010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":8.0920000000000005,"5min":8.8539999999999992,"15min":11.156000000000001},"min":{"1min":5.0110000000000001,"5min":4.5439999999999996,"15min":4.5439999999999996},"max":{"1min":11.35,"5min":35.052,"15min":35.817999999999998},"last":4.532},{"interface":"front","average":{"1min":7.8849999999999998,"5min":9.2420000000000009,"15min":11.247},"min":{"1min":4.6500000000000004,"5min":4.6500000000000004,"15min":4.6500000000000004},"max":{"1min":16.132999999999999,"5min":35.091999999999999,"15min":35.091999999999999},"last":9.9399999999999995}]},{"osd":1,"last update":"Fri Aug 30 16:05:29 2024","interfaces":[{"interface":"back","average":{"1min":12.667,"5min":12.869,"15min":11.863},"min":{"1min":7.1399999999999997,"5min":5.6879999999999997,"15min":4.7000000000000002},"max":{"1min":24.204000000000001,"5min":34.347999999999999,"15min":34.347999999999999},"last":11.477},{"interface":"front","average":{"1min":10.877000000000001,"5min":11.917,"15min":12.324},"min":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"max":{"1min":26.788,"5min":35.481000000000002,"15min":35.481000000000002},"last":8.6129999999999995}]},{"osd":2,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":9.6660000000000004,"5min":9.5470000000000006,"15min":9.9280000000000008},"min":{"1min":5.4930000000000003,"5min":5.4279999999999999,"15min":5.4279999999999999},"max":{"1min":30.666,"5min":30.666,"15min":30.666},"last":7.4059999999999997},{"interface":"front","average":{"1min":11.707000000000001,"5min":12.987,"15min":14.148999999999999},"min":{"1min":7.0629999999999997,"5min":7.0629999999999997,"15min":7.0629999999999997},"max":{"1min":21.355,"5min":29.079999999999998,"15min":30.629000000000001},"last":7.0990000000000002}]},{"osd":3,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":11.263,"5min":11.942,"15min":12.234999999999999},"min":{"1min":7.476,"5min":5.8010000000000002,"15min":5.8010000000000002},"max":{"1min":28.315999999999999,"5min":29.715,"15min":29.715},"last":10.932},{"interface":"front","average":{"1min":13.385,"5min":14.148999999999999,"15min":15.195},"min":{"1min":7.8220000000000001,"5min":7.5830000000000002,"15min":7.5830000000000002},"max":{"1min":27.321999999999999,"5min":27.321999999999999,"15min":31.73},"last":10.449}]},{"osd":5,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":14.595000000000001,"5min":13.505000000000001,"15min":11.943},"min":{"1min":9.1199999999999992,"5min":6.8639999999999999,"15min":6.5010000000000003},"max":{"1min":31.565000000000001,"5min":42.143000000000001,"15min":42.143000000000001},"last":15.023999999999999},{"interface":"front","average":{"1min":12.859,"5min":13.661,"15min":14.194000000000001},"min":{"1min":6.5579999999999998,"5min":6.5579999999999998,"15min":6.5579999999999998},"max":{"1min":29.210999999999999,"5min":43.917000000000002,"15min":43.917000000000002},"last":12.792}]},{"osd":6,"last update":"Fri Aug 30 16:03:23 2024","interfaces":[{"interface":"back","average":{"1min":13.481,"5min":14.797000000000001,"15min":15.696999999999999},"min":{"1min":8.0359999999999996,"5min":6.5289999999999999,"15min":6.5289999999999999},"max":{"1min":20.704000000000001,"5min":37.215000000000003,"15min":37.215000000000003},"last":9.4100000000000001},{"interface":"front","average":{"1min":16.030999999999999,"5min":16.199000000000002,"15min":17.021999999999998},"min":{"1min":9.8300000000000001,"5min":8.7870000000000008,"15min":8.7870000000000008},"max":{"1min":24.986999999999998,"5min":37.753999999999998,"15min":37.753999999999998},"last":15.679}]},{"osd":7,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":14.657,"15min":15.507999999999999},"min":{"1min":7.9950000000000001,"5min":7.7130000000000001,"15min":7.7130000000000001},"max":{"1min":21.992999999999999,"5min":33.314,"15min":34.551000000000002},"last":14.345000000000001},{"interface":"front","average":{"1min":12.443,"5min":15.465,"15min":16.873999999999999},"min":{"1min":6.375,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":18.116,"5min":35.701999999999998,"15min":35.701999999999998},"last":13.589}]}]},{"osd":2,"up_from":16,"seq":68719476824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:17 2024","interfaces":[{"interface":"back","average":{"1min":10.505000000000001,"5min":9.4960000000000004,"15min":9.6129999999999995},"min":{"1min":4.1200000000000001,"5min":4.1200000000000001,"15min":3.6709999999999998},"max":{"1min":25.526,"5min":29.303000000000001,"15min":30.366},"last":6.5629999999999997},{"interface":"front","average":{"1min":11.006,"5min":9.6720000000000006,"15min":10.569000000000001},"min":{"1min":5.4009999999999998,"5min":4.5839999999999996,"15min":3.4900000000000002},"max":{"1min":25.300000000000001,"5min":27.916,"15min":45.604999999999997},"last":6.9059999999999997}]},{"osd":1,"last update":"Fri Aug 30 16:05:33 2024","interfaces":[{"interface":"back","average":{"1min":12.826000000000001,"5min":13.023999999999999,"15min":13.119999999999999},"min":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":41.786999999999999,"5min":41.786999999999999,"15min":46.460000000000001},"last":10.82},{"interface":"front","average":{"1min":15.141,"5min":11.379,"15min":12.558999999999999},"min":{"1min":6.3650000000000002,"5min":5.944,"15min":5.944},"max":{"1min":67.332999999999998,"5min":67.332999999999998,"15min":67.332999999999998},"last":9.032}]},{"osd":3,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":13.191000000000001,"5min":13.573,"15min":16.579000000000001},"min":{"1min":5.7850000000000001,"5min":5.5179999999999998,"15min":5.5179999999999998},"max":{"1min":31.547999999999998,"5min":71.475999999999999,"15min":71.475999999999999},"last":10.074},{"interface":"front","average":{"1min":14.112,"5min":13.813000000000001,"15min":18.425000000000001},"min":{"1min":6.218,"5min":6.218,"15min":6.218},"max":{"1min":40.046999999999997,"5min":40.046999999999997,"15min":50.564},"last":11.938000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":15.548999999999999,"5min":14.409000000000001,"15min":17.457000000000001},"min":{"1min":8.4000000000000004,"5min":7.0970000000000004,"15min":7.0970000000000004},"max":{"1min":28.899000000000001,"5min":33.892000000000003,"15min":48.673999999999999},"last":13.084},{"interface":"front","average":{"1min":13.239000000000001,"5min":14.494,"15min":15.609999999999999},"min":{"1min":7.1879999999999997,"5min":6.7640000000000002,"15min":6.7640000000000002},"max":{"1min":27.448,"5min":67.054000000000002,"15min":67.054000000000002},"last":14.407999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:05:17 2024","interfaces":[{"interface":"back","average":{"1min":13.17,"5min":15.824,"15min":16.684999999999999},"min":{"1min":7.976,"5min":7.0259999999999998,"15min":4.7199999999999998},"max":{"1min":22.773,"5min":69.411000000000001,"15min":69.411000000000001},"last":17.004000000000001},{"interface":"front","average":{"1min":14.057,"5min":16.693000000000001,"15min":18.975000000000001},"min":{"1min":9.5449999999999999,"5min":8.6910000000000007,"15min":5.7460000000000004},"max":{"1min":24.228999999999999,"5min":56.152000000000001,"15min":56.917999999999999},"last":12.577999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:27 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":16.623000000000001,"15min":18.198},"min":{"1min":10.302,"5min":7.4690000000000003,"15min":7.4690000000000003},"max":{"1min":23.317,"5min":72.75,"15min":72.75},"last":13.609},{"interface":"front","average":{"1min":19.774000000000001,"5min":18.396999999999998,"15min":20.119},"min":{"1min":9.8879999999999999,"5min":8.1579999999999995,"15min":8.1579999999999995},"max":{"1min":70.462999999999994,"5min":74.119,"15min":74.119},"last":18.692}]},{"osd":7,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":13.157,"5min":17.632000000000001,"15min":20.654},"min":{"1min":8.3160000000000007,"5min":8.2639999999999993,"15min":8.2639999999999993},"max":{"1min":21.460000000000001,"5min":137.33000000000001,"15min":137.33000000000001},"last":16.34},{"interface":"front","average":{"1min":15.618,"5min":19.195,"15min":19.751000000000001},"min":{"1min":9.7880000000000003,"5min":7.2770000000000001,"15min":7.2770000000000001},"max":{"1min":30.722999999999999,"5min":136.66399999999999,"15min":136.66399999999999},"last":17.559999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509527,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":7.3120000000000003,"5min":10.007,"15min":9.7769999999999992},"min":{"1min":4.8090000000000002,"5min":4.7489999999999997,"15min":4.7489999999999997},"max":{"1min":14.276,"5min":40.735999999999997,"15min":40.735999999999997},"last":9.7829999999999995},{"interface":"front","average":{"1min":10.282,"5min":11.349,"15min":11.352},"min":{"1min":5.0069999999999997,"5min":4.383,"15min":4.383},"max":{"1min":25.077999999999999,"5min":42.387,"15min":42.387},"last":5.8369999999999997}]},{"osd":2,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":10.721,"5min":11.506,"15min":11.308},"min":{"1min":6.3899999999999997,"5min":5.3689999999999998,"15min":3.3370000000000002},"max":{"1min":24.170999999999999,"5min":46.768999999999998,"15min":46.768999999999998},"last":6.9020000000000001},{"interface":"front","average":{"1min":13.676,"5min":13.199999999999999,"15min":14.022},"min":{"1min":6.8860000000000001,"5min":6.2249999999999996,"15min":5.702},"max":{"1min":31.361000000000001,"5min":43.921999999999997,"15min":43.921999999999997},"last":12.807}]},{"osd":3,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":11.460000000000001,"5min":13.585000000000001,"15min":21.405000000000001},"min":{"1min":7.702,"5min":6.2839999999999998,"15min":6.2839999999999998},"max":{"1min":29.998999999999999,"5min":38.219000000000001,"15min":200.155},"last":14.454000000000001},{"interface":"front","average":{"1min":15.455,"5min":16.989000000000001,"15min":26.460999999999999},"min":{"1min":10.507999999999999,"5min":7.0940000000000003,"15min":7.0940000000000003},"max":{"1min":28.146000000000001,"5min":44.229999999999997,"15min":267.03100000000001},"last":17.126000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.647,"5min":13.015000000000001,"15min":15.007},"min":{"1min":6.891,"5min":6.0970000000000004,"15min":6.0970000000000004},"max":{"1min":14.891,"5min":31.611999999999998,"15min":56.200000000000003},"last":10.116},{"interface":"front","average":{"1min":13.050000000000001,"5min":15.571,"15min":15.683999999999999},"min":{"1min":9.3930000000000007,"5min":9.298,"15min":8.609},"max":{"1min":16.626999999999999,"5min":46.875999999999998,"15min":48.534999999999997},"last":16.398}]},{"osd":5,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":14.43,"5min":15.682,"15min":16.082999999999998},"min":{"1min":9.8059999999999992,"5min":5.9199999999999999,"15min":5.6920000000000002},"max":{"1min":40.006,"5min":43.308,"15min":43.308},"last":14.856},{"interface":"front","average":{"1min":16.951000000000001,"5min":17.693000000000001,"15min":17.853999999999999},"min":{"1min":9.9100000000000001,"5min":9.2149999999999999,"15min":4.75},"max":{"1min":49.575000000000003,"5min":49.575000000000003,"15min":49.575000000000003},"last":18.652999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":13.808999999999999,"5min":14.292,"15min":14.385999999999999},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":39.460999999999999,"5min":59.328000000000003,"15min":59.328000000000003},"last":15.618},{"interface":"front","average":{"1min":19.225000000000001,"5min":18.331,"15min":18.533999999999999},"min":{"1min":10.781000000000001,"5min":10.005000000000001,"15min":10.005000000000001},"max":{"1min":46.237000000000002,"5min":58.567,"15min":58.567},"last":11.281000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":18.472000000000001,"5min":19.276,"15min":21.135999999999999},"min":{"1min":11.045,"5min":10.632999999999999,"15min":9.1180000000000003},"max":{"1min":57.804000000000002,"5min":57.804000000000002,"15min":59.515999999999998},"last":20.552},{"interface":"front","average":{"1min":11.272,"5min":13.384,"15min":14.976000000000001},"min":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":7.0590000000000002},"max":{"1min":14.231999999999999,"5min":37.113,"15min":58.807000000000002},"last":17.818000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574935,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":658493,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:20 2024","interfaces":[{"interface":"back","average":{"1min":6.2510000000000003,"5min":8.2520000000000007,"15min":12.516999999999999},"min":{"1min":4.0300000000000002,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":13.52,"5min":53.558999999999997,"15min":111.313},"last":9.7219999999999995},{"interface":"front","average":{"1min":8.1809999999999992,"5min":9.4659999999999993,"15min":13.936},"min":{"1min":4.266,"5min":4.266,"15min":4.266},"max":{"1min":14.064,"5min":52.691000000000003,"15min":107.937},"last":7.2809999999999997}]},{"osd":1,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":9.3030000000000008,"5min":10.489000000000001,"15min":12.541},"min":{"1min":5.4539999999999997,"5min":4.657,"15min":4.657},"max":{"1min":16.811,"5min":34.545999999999999,"15min":46.988},"last":6.0279999999999996},{"interface":"front","average":{"1min":10.688000000000001,"5min":12.9,"15min":15.188000000000001},"min":{"1min":7.5179999999999998,"5min":6.4710000000000001,"15min":6.4710000000000001},"max":{"1min":15.455,"5min":33.203000000000003,"15min":47.661000000000001},"last":11.407999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:05:20 2024","interfaces":[{"interface":"back","average":{"1min":13.909000000000001,"5min":15.346,"15min":17.995000000000001},"min":{"1min":8.5630000000000006,"5min":6.9569999999999999,"15min":4.7220000000000004},"max":{"1min":31.667000000000002,"5min":41.549999999999997,"15min":84.317999999999998},"last":10.867000000000001},{"interface":"front","average":{"1min":11.781000000000001,"5min":13.901,"15min":16.207000000000001},"min":{"1min":6.9829999999999997,"5min":6.2770000000000001,"15min":6.0279999999999996},"max":{"1min":29.847999999999999,"5min":43.314999999999998,"15min":99.884},"last":6.7290000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":12.699,"5min":16.114000000000001,"15min":16.768999999999998},"min":{"1min":9.6899999999999995,"5min":6.2149999999999999,"15min":6.2149999999999999},"max":{"1min":18.687000000000001,"5min":117.346,"15min":117.346},"last":10.335000000000001},{"interface":"front","average":{"1min":12.143000000000001,"5min":14.827999999999999,"15min":15.944000000000001},"min":{"1min":6.5270000000000001,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":19.370999999999999,"5min":119.23099999999999,"15min":119.23099999999999},"last":15.724}]},{"osd":4,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":12.651999999999999,"5min":15.768000000000001,"15min":17.32},"min":{"1min":6.5599999999999996,"5min":6.5599999999999996,"15min":6.5599999999999996},"max":{"1min":20.619,"5min":33.582000000000001,"15min":38.095999999999997},"last":12.704000000000001},{"interface":"front","average":{"1min":12.183999999999999,"5min":14.702,"15min":16.603999999999999},"min":{"1min":6.71,"5min":6.71,"15min":6.71},"max":{"1min":20.678999999999998,"5min":31.367000000000001,"15min":40.649000000000001},"last":13.302}]},{"osd":6,"last update":"Fri Aug 30 16:03:26 2024","interfaces":[{"interface":"back","average":{"1min":9.516,"5min":10.686,"15min":11.025},"min":{"1min":6.7809999999999997,"5min":6.367,"15min":6.367},"max":{"1min":15.27,"5min":26.344999999999999,"15min":26.344999999999999},"last":7.8949999999999996},{"interface":"front","average":{"1min":19.596,"5min":17.696999999999999,"15min":17.396000000000001},"min":{"1min":11.433,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":58.052999999999997,"5min":58.052999999999997,"15min":58.052999999999997},"last":14.106}]},{"osd":7,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":13.638,"5min":15.441000000000001,"15min":17.472999999999999},"min":{"1min":7.8490000000000002,"5min":7.8490000000000002,"15min":7.8490000000000002},"max":{"1min":25.056999999999999,"5min":38.180999999999997,"15min":43.604999999999997},"last":15.029999999999999},{"interface":"front","average":{"1min":9.7629999999999999,"5min":11.205,"15min":16.343},"min":{"1min":5.3579999999999997,"5min":5.3579999999999997,"15min":5.3579999999999997},"max":{"1min":22.509,"5min":31.547999999999998,"15min":88.278000000000006},"last":9.2070000000000007}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:06:08.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-30T16:06:08.762 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:06:08.762 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":130,"fsid":"9d72e2e2-0554-42af-8e62-9b1a3a804112","created":"2024-08-30T15:58:20.129410+0000","modified":"2024-08-30T16:05:58.319477+0000","last_up_change":"2024-08-30T16:05:24.091073+0000","last_in_change":"2024-08-30T16:05:38.025676+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-30T15:59:08.355540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-30T15:59:21.696796+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":2.630000114440918,"raw_score_stable":2.630000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9c399d4c-9a92-4ed9-a77b-af0dd0ff02d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6812","nonce":1957399494}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6813","nonce":1957399494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6815","nonce":1957399494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6814","nonce":1957399494}]},"public_addr":"172.21.15.50:6812/1957399494","cluster_addr":"172.21.15.50:6813/1957399494","heartbeat_back_addr":"172.21.15.50:6815/1957399494","heartbeat_front_addr":"172.21.15.50:6814/1957399494","state":["exists","up"]},{"osd":1,"uuid":"0fe11a1c-4326-4bfd-a2a4-bff14023f301","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6808","nonce":3208178756}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6809","nonce":3208178756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6811","nonce":3208178756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6810","nonce":3208178756}]},"public_addr":"172.21.15.50:6808/3208178756","cluster_addr":"172.21.15.50:6809/3208178756","heartbeat_back_addr":"172.21.15.50:6811/3208178756","heartbeat_front_addr":"172.21.15.50:6810/3208178756","state":["exists","up"]},{"osd":2,"uuid":"a9d5354b-3600-41ed-80c5-a3439da7aca6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6804","nonce":287018800}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6805","nonce":287018800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6807","nonce":287018800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6806","nonce":287018800}]},"public_addr":"172.21.15.50:6804/287018800","cluster_addr":"172.21.15.50:6805/287018800","heartbeat_back_addr":"172.21.15.50:6807/287018800","heartbeat_front_addr":"172.21.15.50:6806/287018800","state":["exists","up"]},{"osd":3,"uuid":"78bcc398-a7e4-4efc-9836-1dfb94407c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6800","nonce":1140717714}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6801","nonce":1140717714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6803","nonce":1140717714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.50:6802","nonce":1140717714}]},"public_addr":"172.21.15.50:6800/1140717714","cluster_addr":"172.21.15.50:6801/1140717714","heartbeat_back_addr":"172.21.15.50:6803/1140717714","heartbeat_front_addr":"172.21.15.50:6802/1140717714","state":["exists","up"]},{"osd":4,"uuid":"53a1060b-7985-4c2c-820e-64b158056113","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6805","nonce":1468073345}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6806","nonce":1468073345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6808","nonce":1468073345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6807","nonce":1468073345}]},"public_addr":"172.21.15.94:6805/1468073345","cluster_addr":"172.21.15.94:6806/1468073345","heartbeat_back_addr":"172.21.15.94:6808/1468073345","heartbeat_front_addr":"172.21.15.94:6807/1468073345","state":["exists","up"]},{"osd":5,"uuid":"853a03f7-227d-4f98-9e5c-c29896820746","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6809","nonce":3876025616}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6810","nonce":3876025616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6812","nonce":3876025616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6811","nonce":3876025616}]},"public_addr":"172.21.15.94:6809/3876025616","cluster_addr":"172.21.15.94:6810/3876025616","heartbeat_back_addr":"172.21.15.94:6812/3876025616","heartbeat_front_addr":"172.21.15.94:6811/3876025616","state":["exists","up"]},{"osd":6,"uuid":"bc978060-bf28-4027-8bba-7d6d8659bfc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":74,"up_from":105,"up_thru":0,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6801","nonce":1984944988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6802","nonce":1984944988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6804","nonce":1984944988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6803","nonce":1984944988}]},"public_addr":"172.21.15.94:6801/1984944988","cluster_addr":"172.21.15.94:6802/1984944988","heartbeat_back_addr":"172.21.15.94:6804/1984944988","heartbeat_front_addr":"172.21.15.94:6803/1984944988","state":["exists","up"]},{"osd":7,"uuid":"45df5c6f-b124-46d1-9efb-62b78bc584ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6813","nonce":68721277}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6814","nonce":68721277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6816","nonce":68721277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.94:6815","nonce":68721277}]},"public_addr":"172.21.15.94:6813/68721277","cluster_addr":"172.21.15.94:6814/68721277","heartbeat_back_addr":"172.21.15.94:6816/68721277","heartbeat_front_addr":"172.21.15.94:6815/68721277","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.911895+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.786543+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:04.197952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.680847+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:01.937600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:02.082761+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.401025+0000","dead_epoch":74},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T15:59:03.129371+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.3","osds":[7,0]},{"pgid":"2.5","osds":[2,5]},{"pgid":"2.7","osds":[7,0]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T16:06:09.776 INFO:tasks.ceph:Scrubbing osd.0 2024-08-30T16:06:09.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-08-30T16:06:09.975 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:06:09.975 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-30T16:06:09.975 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:06:09.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-08-30T16:06:10.824 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 0 to deep-scrub 2024-08-30T16:06:10.838 INFO:tasks.ceph:Scrubbing osd.1 2024-08-30T16:06:10.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-08-30T16:06:11.032 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:06:11.032 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-30T16:06:11.033 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:06:11.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-08-30T16:06:11.576 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 1 to deep-scrub 2024-08-30T16:06:11.590 INFO:tasks.ceph:Scrubbing osd.2 2024-08-30T16:06:11.590 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-08-30T16:06:11.782 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:06:11.783 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-30T16:06:11.783 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:06:11.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-08-30T16:06:12.339 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 2 to deep-scrub 2024-08-30T16:06:12.353 INFO:tasks.ceph:Scrubbing osd.3 2024-08-30T16:06:12.353 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-08-30T16:06:12.544 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:06:12.545 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-30T16:06:12.545 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:06:12.555 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-08-30T16:06:13.142 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 3 to deep-scrub 2024-08-30T16:06:13.159 INFO:tasks.ceph:Scrubbing osd.4 2024-08-30T16:06:13.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-08-30T16:06:13.349 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:06:13.349 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-30T16:06:13.349 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:06:13.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-08-30T16:06:13.787 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:36.392 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:13.789 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:36.395 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:13.808 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:36.414 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:13.838 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:35.375 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:13.840 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:35.377 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:13.841 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:35.001 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:13.843 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:35.003 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:13.858 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:35.395 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:13.862 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:35.022 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:13.916 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 4 to deep-scrub 2024-08-30T16:06:13.929 INFO:tasks.ceph:Scrubbing osd.5 2024-08-30T16:06:13.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-08-30T16:06:14.254 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:06:14.254 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-30T16:06:14.254 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:06:14.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-08-30T16:06:14.802 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 5 to deep-scrub 2024-08-30T16:06:14.816 INFO:tasks.ceph:Scrubbing osd.6 2024-08-30T16:06:14.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-08-30T16:06:15.022 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:06:15.022 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-08-30T16:06:15.022 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:06:15.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-08-30T16:06:15.628 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 6 to deep-scrub 2024-08-30T16:06:15.642 INFO:tasks.ceph:Scrubbing osd.7 2024-08-30T16:06:15.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-08-30T16:06:15.827 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-08-30T16:06:15.827 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-30T16:06:15.827 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-08-30T16:06:15.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-08-30T16:06:16.396 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 7 to deep-scrub 2024-08-30T16:06:16.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:06:16.979 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:06:16.979 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:06:16.990 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-08-30T16:06:15.939997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1591398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222156,"kb_used_data":5480,"kb_used_omap":15,"kb_used_meta":216560,"kb_avail":754752564,"statfs":{"total":773094113280,"available":772866625536,"internally_reserved":0,"allocated":5611520,"data_stored":2880942,"data_compressed":1558,"data_compressed_allocated":49152,"data_compressed_original":786432,"omap_allocated":15591,"internal_metadata":221758233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":171,"apply_latency_ms":171,"commit_latency_ns":171000000,"apply_latency_ns":171000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001747"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.467046+0000","last_change":"2024-08-30T16:04:11.379068+0000","last_active":"2024-08-30T16:05:57.467046+0000","last_peered":"2024-08-30T16:05:57.467046+0000","last_clean":"2024-08-30T16:05:57.467046+0000","last_became_active":"2024-08-30T16:04:11.370524+0000","last_became_peered":"2024-08-30T16:04:11.370524+0000","last_unstale":"2024-08-30T16:05:57.467046+0000","last_undegraded":"2024-08-30T16:05:57.467046+0000","last_fullsized":"2024-08-30T16:05:57.467046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:58:06.441243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.022669+0000","last_change":"2024-08-30T16:04:38.088583+0000","last_active":"2024-08-30T16:05:58.022669+0000","last_peered":"2024-08-30T16:05:58.022669+0000","last_clean":"2024-08-30T16:05:58.022669+0000","last_became_active":"2024-08-30T16:04:38.082821+0000","last_became_peered":"2024-08-30T16:04:38.082821+0000","last_unstale":"2024-08-30T16:05:58.022669+0000","last_undegraded":"2024-08-30T16:05:58.022669+0000","last_fullsized":"2024-08-30T16:05:58.022669+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:40:53.077427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2024-08-30T16:06:12.353896+0000","last_change":"2024-08-30T16:06:12.853908+0000","last_active":"2024-08-30T16:06:12.353896+0000","last_peered":"2024-08-30T16:06:12.353896+0000","last_clean":"2024-08-30T16:06:12.353896+0000","last_became_active":"2024-08-30T16:05:38.361240+0000","last_became_peered":"2024-08-30T16:05:38.361240+0000","last_unstale":"2024-08-30T16:06:12.353896+0000","last_undegraded":"2024-08-30T16:06:12.353896+0000","last_fullsized":"2024-08-30T16:06:12.353896+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.259961+0000","last_change":"2024-08-30T16:05:39.779631+0000","last_active":"2024-08-30T16:05:57.259961+0000","last_peered":"2024-08-30T16:05:57.259961+0000","last_clean":"2024-08-30T16:05:57.259961+0000","last_became_active":"2024-08-30T16:05:39.773479+0000","last_became_peered":"2024-08-30T16:05:39.773479+0000","last_unstale":"2024-08-30T16:05:57.259961+0000","last_undegraded":"2024-08-30T16:05:57.259961+0000","last_fullsized":"2024-08-30T16:05:57.259961+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:29:29.455454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":144,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.469405+0000","last_change":"2024-08-30T16:05:40.031358+0000","last_active":"2024-08-30T16:05:58.469405+0000","last_peered":"2024-08-30T16:05:58.469405+0000","last_clean":"2024-08-30T16:05:58.469405+0000","last_became_active":"2024-08-30T16:05:39.776453+0000","last_became_peered":"2024-08-30T16:05:39.776453+0000","last_unstale":"2024-08-30T16:05:58.469405+0000","last_undegraded":"2024-08-30T16:05:58.469405+0000","last_fullsized":"2024-08-30T16:05:58.469405+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:57.462397+0000","last_change":"2024-08-30T16:05:38.365381+0000","last_active":"2024-08-30T16:05:57.462397+0000","last_peered":"2024-08-30T16:05:57.462397+0000","last_clean":"2024-08-30T16:05:57.462397+0000","last_became_active":"2024-08-30T16:05:38.358584+0000","last_became_peered":"2024-08-30T16:05:38.358584+0000","last_unstale":"2024-08-30T16:05:57.462397+0000","last_undegraded":"2024-08-30T16:05:57.462397+0000","last_fullsized":"2024-08-30T16:05:57.462397+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:28:34.896593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":142,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.467052+0000","last_change":"2024-08-30T16:05:38.364964+0000","last_active":"2024-08-30T16:05:58.467052+0000","last_peered":"2024-08-30T16:05:58.467052+0000","last_clean":"2024-08-30T16:05:58.467052+0000","last_became_active":"2024-08-30T16:05:38.358598+0000","last_became_peered":"2024-08-30T16:05:38.358598+0000","last_unstale":"2024-08-30T16:05:58.467052+0000","last_undegraded":"2024-08-30T16:05:58.467052+0000","last_fullsized":"2024-08-30T16:05:58.467052+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"25'1","reported_seq":61,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T16:05:58.904323+0000","last_change":"2024-08-30T16:04:08.039936+0000","last_active":"2024-08-30T16:05:58.904323+0000","last_peered":"2024-08-30T16:05:58.904323+0000","last_clean":"2024-08-30T16:05:58.904323+0000","last_became_active":"2024-08-30T16:04:07.760364+0000","last_became_peered":"2024-08-30T16:04:07.760364+0000","last_unstale":"2024-08-30T16:05:58.904323+0000","last_undegraded":"2024-08-30T16:05:58.904323+0000","last_fullsized":"2024-08-30T16:05:58.904323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:22.564097+0000","last_clean_scrub_stamp":"2024-08-30T15:59:22.564097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:13:37.016725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"130'78","reported_seq":168,"reported_epoch":130,"state":"active+clean+scrubbing+deep","last_fresh":"2024-08-30T16:06:15.376918+0000","last_change":"2024-08-30T16:06:15.079374+0000","last_active":"2024-08-30T16:06:15.376918+0000","last_peered":"2024-08-30T16:06:15.376918+0000","last_clean":"2024-08-30T16:06:15.376918+0000","last_became_active":"2024-08-30T16:05:38.360837+0000","last_became_peered":"2024-08-30T16:05:38.360837+0000","last_unstale":"2024-08-30T16:06:15.376918+0000","last_undegraded":"2024-08-30T16:06:15.376918+0000","last_fullsized":"2024-08-30T16:06:15.376918+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T15:59:09.163382+0000","last_clean_scrub_stamp":"2024-08-30T15:59:09.163382+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1591360,"data_compressed":3116,"data_compressed_allocated":98304,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280605,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27732,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344108,"statfs":{"total":96636764160,"available":96608366592,"internally_reserved":0,"allocated":724992,"data_stored":658493,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":162,"apply_latency_ms":162,"commit_latency_ns":162000000,"apply_latency_ns":162000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.896000000000001,"5min":11.542,"15min":13.461},"min":{"1min":5.5519999999999996,"5min":4.8049999999999997,"15min":4.8049999999999997},"max":{"1min":25.129999999999999,"5min":51.027000000000001,"15min":57.177999999999997},"last":15.929},{"interface":"front","average":{"1min":9.5020000000000007,"5min":11.342000000000001,"15min":12.673999999999999},"min":{"1min":4.5759999999999996,"5min":4.5759999999999996,"15min":4.5759999999999996},"max":{"1min":16.521000000000001,"5min":54.682000000000002,"15min":58.024999999999999},"last":5.8840000000000003}]},{"osd":2,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.157,"5min":10.984999999999999,"15min":10.614000000000001},"min":{"1min":6.0880000000000001,"5min":5.5780000000000003,"15min":5.5780000000000003},"max":{"1min":15.81,"5min":73.399000000000001,"15min":73.399000000000001},"last":12.462},{"interface":"front","average":{"1min":13.478,"5min":14.470000000000001,"15min":14.875},"min":{"1min":6.5640000000000001,"5min":6.0049999999999999,"15min":6.0049999999999999},"max":{"1min":30.704999999999998,"5min":75.120000000000005,"15min":75.120000000000005},"last":84.010999999999996}]},{"osd":3,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":13.753,"5min":15.361000000000001,"15min":16.896000000000001},"min":{"1min":9.1199999999999992,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":29.506,"5min":66.869,"15min":66.869},"last":210.381},{"interface":"front","average":{"1min":10.523999999999999,"5min":13.089,"15min":14.315},"min":{"1min":5.8959999999999999,"5min":5.8959999999999999,"15min":5.8959999999999999},"max":{"1min":16.196000000000002,"5min":104.38200000000001,"15min":104.38200000000001},"last":171.33000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":11.037000000000001,"5min":14.484999999999999,"15min":14.68},"min":{"1min":7.5590000000000002,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":18.202999999999999,"5min":66.340999999999994,"15min":66.340999999999994},"last":11.856},{"interface":"front","average":{"1min":9.5540000000000003,"5min":12.513999999999999,"15min":12.881},"min":{"1min":6.774,"5min":6.774,"15min":6.774},"max":{"1min":14.335000000000001,"5min":29.818999999999999,"15min":29.818999999999999},"last":7.1820000000000004}]},{"osd":5,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":13.954000000000001,"5min":15.705,"15min":16.510999999999999},"min":{"1min":7.21,"5min":7.21,"15min":7.21},"max":{"1min":31.545000000000002,"5min":55.784999999999997,"15min":55.784999999999997},"last":79.052000000000007},{"interface":"front","average":{"1min":11.759,"5min":17.265999999999998,"15min":18.946000000000002},"min":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"max":{"1min":16.829999999999998,"5min":68.004999999999995,"15min":68.004999999999995},"last":8.7560000000000002}]},{"osd":6,"last update":"Fri Aug 30 16:03:30 2024","interfaces":[{"interface":"back","average":{"1min":15.708,"5min":16.609000000000002,"15min":17.382000000000001},"min":{"1min":10.180999999999999,"5min":9.1929999999999996,"15min":9.1929999999999996},"max":{"1min":27.353999999999999,"5min":58.201999999999998,"15min":58.201999999999998},"last":208.61199999999999},{"interface":"front","average":{"1min":16.324999999999999,"5min":16.748000000000001,"15min":17.001000000000001},"min":{"1min":9.6980000000000004,"5min":8.8390000000000004,"15min":8.8390000000000004},"max":{"1min":26.257999999999999,"5min":37.253,"15min":37.253},"last":13.567}]},{"osd":7,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":16.931000000000001,"5min":17.739999999999998,"15min":19.425999999999998},"min":{"1min":10.352,"5min":8.2050000000000001,"15min":8.2050000000000001},"max":{"1min":34.345999999999997,"5min":68.950000000000003,"15min":68.950000000000003},"last":209.614},{"interface":"front","average":{"1min":13.039999999999999,"5min":16.266999999999999,"15min":18.443999999999999},"min":{"1min":9.4529999999999994,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":17.625,"5min":36.186,"15min":59.079000000000001},"last":83.188999999999993}]}]},{"osd":7,"up_from":21,"seq":90194313308,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":7.9219999999999997,"5min":8.6739999999999995,"15min":8.8670000000000009},"min":{"1min":4.3789999999999996,"5min":4.3789999999999996,"15min":4.3789999999999996},"max":{"1min":20.73,"5min":25.759,"15min":30.600999999999999},"last":12.226000000000001},{"interface":"front","average":{"1min":7.1589999999999998,"5min":8.8819999999999997,"15min":9.0419999999999998},"min":{"1min":4.8460000000000001,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":12.57,"5min":28.077000000000002,"15min":28.077000000000002},"last":11.019}]},{"osd":1,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":11.804,"5min":12.818,"15min":11.496},"min":{"1min":7.0590000000000002,"5min":5.7009999999999996,"15min":5.7009999999999996},"max":{"1min":18.989999999999998,"5min":83.581000000000003,"15min":83.581000000000003},"last":10.472},{"interface":"front","average":{"1min":9.2690000000000001,"5min":11.781000000000001,"15min":12.198},"min":{"1min":5.6230000000000002,"5min":5.6230000000000002,"15min":5.6230000000000002},"max":{"1min":19.524999999999999,"5min":81.722999999999999,"15min":81.722999999999999},"last":15.99}]},{"osd":2,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":14.83,"5min":13.890000000000001,"15min":13.689},"min":{"1min":8.5039999999999996,"5min":5.4850000000000003,"15min":5.4850000000000003},"max":{"1min":25.885999999999999,"5min":42.506,"15min":42.506},"last":86.873000000000005},{"interface":"front","average":{"1min":13.273999999999999,"5min":13.69,"15min":16.242000000000001},"min":{"1min":6.8029999999999999,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":24.309000000000001,"5min":43.484999999999999,"15min":43.484999999999999},"last":89.247}]},{"osd":3,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":15.515000000000001,"5min":15.986000000000001,"15min":16.391999999999999},"min":{"1min":7.9299999999999997,"5min":7.9299999999999997,"15min":7.9299999999999997},"max":{"1min":54.076999999999998,"5min":65.739999999999995,"15min":65.739999999999995},"last":13.304},{"interface":"front","average":{"1min":12.406000000000001,"5min":13.285,"15min":12.871},"min":{"1min":5.6070000000000002,"5min":5.6070000000000002,"15min":5.6070000000000002},"max":{"1min":55.978000000000002,"5min":64.980000000000004,"15min":64.980000000000004},"last":15.051}]},{"osd":4,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":13.824,"15min":12.827},"min":{"1min":6.4880000000000004,"5min":6.4880000000000004,"15min":6.4880000000000004},"max":{"1min":23.213999999999999,"5min":41.631,"15min":41.631},"last":17.509},{"interface":"front","average":{"1min":14.590999999999999,"5min":15.433,"15min":17.003},"min":{"1min":9.2789999999999999,"5min":8.8729999999999993,"15min":8.8729999999999993},"max":{"1min":25.149999999999999,"5min":32.526000000000003,"15min":37.393000000000001},"last":8.5920000000000005}]},{"osd":5,"last update":"Fri Aug 30 16:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.68,"5min":12.377000000000001,"15min":11.679},"min":{"1min":6.8259999999999996,"5min":6.2309999999999999,"15min":6.2309999999999999},"max":{"1min":37.372999999999998,"5min":37.372999999999998,"15min":37.372999999999998},"last":14.159000000000001},{"interface":"front","average":{"1min":16.558,"5min":15.769,"15min":16.600000000000001},"min":{"1min":9.0429999999999993,"5min":8.2870000000000008,"15min":8.2870000000000008},"max":{"1min":38.484000000000002,"5min":38.618000000000002,"15min":38.618000000000002},"last":18.448}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":13.326000000000001,"5min":13.771000000000001,"15min":13.208},"min":{"1min":8.0060000000000002,"5min":6.3230000000000004,"15min":6.3230000000000004},"max":{"1min":19.231000000000002,"5min":42.829999999999998,"15min":42.829999999999998},"last":16.745000000000001},{"interface":"front","average":{"1min":15.842000000000001,"5min":17.606000000000002,"15min":18.088999999999999},"min":{"1min":11.712,"5min":10.749000000000001,"15min":10.749000000000001},"max":{"1min":22.695,"5min":44.491999999999997,"15min":44.491999999999997},"last":19.084}]}]},{"osd":6,"up_from":105,"seq":450971566093,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":680,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":696320,"data_stored":260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":28176046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7089999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":88.594999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.376000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.339}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.542999999999999}]}]},{"osd":3,"up_from":21,"seq":90194313308,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":696320,"data_stored":260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":9.4120000000000008,"5min":9.5199999999999996,"15min":10.925000000000001},"min":{"1min":5.0830000000000002,"5min":4.3810000000000002,"15min":4.29},"max":{"1min":18.713000000000001,"5min":46.146999999999998,"15min":50.500999999999998},"last":6.0350000000000001},{"interface":"front","average":{"1min":9.7110000000000003,"5min":10.157,"15min":13.363},"min":{"1min":5.9279999999999999,"5min":5.1360000000000001,"15min":5.1360000000000001},"max":{"1min":20.637,"5min":36.720999999999997,"15min":51.965000000000003},"last":8.7309999999999999}]},{"osd":1,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":12.821999999999999,"5min":13.605,"15min":17.742999999999999},"min":{"1min":7.7549999999999999,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":20.864999999999998,"5min":44.862000000000002,"15min":137.56700000000001},"last":9.4369999999999994},{"interface":"front","average":{"1min":10.217000000000001,"5min":12.255000000000001,"15min":15.741},"min":{"1min":6.7839999999999998,"5min":5.0060000000000002,"15min":5.0060000000000002},"max":{"1min":27.032,"5min":47.530999999999999,"15min":136.321},"last":7.9610000000000003}]},{"osd":2,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":11.897,"5min":12.744999999999999,"15min":15.519},"min":{"1min":5.9109999999999996,"5min":5.9109999999999996,"15min":5.9109999999999996},"max":{"1min":26.527000000000001,"5min":43.869999999999997,"15min":73.831000000000003},"last":12.393000000000001},{"interface":"front","average":{"1min":11.170999999999999,"5min":11.682,"15min":14.923},"min":{"1min":6.984,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":16.887,"5min":26.074999999999999,"15min":71.831999999999994},"last":10.393000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":12.803000000000001,"5min":14.353,"15min":26.016999999999999},"min":{"1min":7.2350000000000003,"5min":6.9199999999999999,"15min":6.9199999999999999},"max":{"1min":25.645,"5min":43.235999999999997,"15min":342.34500000000003},"last":13.93},{"interface":"front","average":{"1min":12.157,"5min":14.849,"15min":19.047000000000001},"min":{"1min":7.1050000000000004,"5min":6.7859999999999996,"15min":6.7859999999999996},"max":{"1min":23.939,"5min":63.633000000000003,"15min":127.857},"last":11.705}]},{"osd":5,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":10.967000000000001,"5min":14.295,"15min":15.500999999999999},"min":{"1min":7.1200000000000001,"5min":6.5,"15min":6.5},"max":{"1min":16.876000000000001,"5min":58.715000000000003,"15min":58.715000000000003},"last":17.870999999999999},{"interface":"front","average":{"1min":14.762,"5min":16.952000000000002,"15min":18.649000000000001},"min":{"1min":9.1549999999999994,"5min":7.5869999999999997,"15min":7.2370000000000001},"max":{"1min":29.803999999999998,"5min":58.212000000000003,"15min":58.212000000000003},"last":14.481999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:25 2024","interfaces":[{"interface":"back","average":{"1min":16.295000000000002,"5min":23.492000000000001,"15min":30.053999999999998},"min":{"1min":11.096,"5min":8.6630000000000003,"15min":8.6630000000000003},"max":{"1min":25.298999999999999,"5min":388.96100000000001,"15min":388.96100000000001},"last":18.850999999999999},{"interface":"front","average":{"1min":13.961,"5min":24.18,"15min":32.835999999999999},"min":{"1min":9.4079999999999995,"5min":8.5519999999999996,"15min":8.5519999999999996},"max":{"1min":20.439,"5min":448.69200000000001,"15min":448.69200000000001},"last":16.190999999999999}]},{"osd":7,"last update":"Fri Aug 30 16:05:26 2024","interfaces":[{"interface":"back","average":{"1min":17.667000000000002,"5min":18.959,"15min":24.385000000000002},"min":{"1min":11.335000000000001,"5min":7.3049999999999997,"15min":7.3049999999999997},"max":{"1min":39.133000000000003,"5min":91.646000000000001,"15min":177.596},"last":17.164999999999999},{"interface":"front","average":{"1min":15.132999999999999,"5min":18.663,"15min":25.454999999999998},"min":{"1min":6.891,"5min":6.891,"15min":6.891},"max":{"1min":35.683,"5min":101.158,"15min":231.03800000000001},"last":15.127000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346011,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":8.0920000000000005,"5min":8.8539999999999992,"15min":11.156000000000001},"min":{"1min":5.0110000000000001,"5min":4.5439999999999996,"15min":4.5439999999999996},"max":{"1min":11.35,"5min":35.052,"15min":35.817999999999998},"last":5.3719999999999999},{"interface":"front","average":{"1min":7.8849999999999998,"5min":9.2420000000000009,"15min":11.247},"min":{"1min":4.6500000000000004,"5min":4.6500000000000004,"15min":4.6500000000000004},"max":{"1min":16.132999999999999,"5min":35.091999999999999,"15min":35.091999999999999},"last":7.9669999999999996}]},{"osd":1,"last update":"Fri Aug 30 16:05:29 2024","interfaces":[{"interface":"back","average":{"1min":12.667,"5min":12.869,"15min":11.863},"min":{"1min":7.1399999999999997,"5min":5.6879999999999997,"15min":4.7000000000000002},"max":{"1min":24.204000000000001,"5min":34.347999999999999,"15min":34.347999999999999},"last":6.165},{"interface":"front","average":{"1min":10.877000000000001,"5min":11.917,"15min":12.324},"min":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":5.4829999999999997},"max":{"1min":26.788,"5min":35.481000000000002,"15min":35.481000000000002},"last":6.8890000000000002}]},{"osd":2,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":9.6660000000000004,"5min":9.5470000000000006,"15min":9.9280000000000008},"min":{"1min":5.4930000000000003,"5min":5.4279999999999999,"15min":5.4279999999999999},"max":{"1min":30.666,"5min":30.666,"15min":30.666},"last":9.2739999999999991},{"interface":"front","average":{"1min":11.707000000000001,"5min":12.987,"15min":14.148999999999999},"min":{"1min":7.0629999999999997,"5min":7.0629999999999997,"15min":7.0629999999999997},"max":{"1min":21.355,"5min":29.079999999999998,"15min":30.629000000000001},"last":13.305}]},{"osd":3,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":11.263,"5min":11.942,"15min":12.234999999999999},"min":{"1min":7.476,"5min":5.8010000000000002,"15min":5.8010000000000002},"max":{"1min":28.315999999999999,"5min":29.715,"15min":29.715},"last":10.066000000000001},{"interface":"front","average":{"1min":13.385,"5min":14.148999999999999,"15min":15.195},"min":{"1min":7.8220000000000001,"5min":7.5830000000000002,"15min":7.5830000000000002},"max":{"1min":27.321999999999999,"5min":27.321999999999999,"15min":31.73},"last":14.417999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":14.595000000000001,"5min":13.505000000000001,"15min":11.943},"min":{"1min":9.1199999999999992,"5min":6.8639999999999999,"15min":6.5010000000000003},"max":{"1min":31.565000000000001,"5min":42.143000000000001,"15min":42.143000000000001},"last":15.880000000000001},{"interface":"front","average":{"1min":12.859,"5min":13.661,"15min":14.194000000000001},"min":{"1min":6.5579999999999998,"5min":6.5579999999999998,"15min":6.5579999999999998},"max":{"1min":29.210999999999999,"5min":43.917000000000002,"15min":43.917000000000002},"last":10.804}]},{"osd":6,"last update":"Fri Aug 30 16:03:23 2024","interfaces":[{"interface":"back","average":{"1min":13.481,"5min":14.797000000000001,"15min":15.696999999999999},"min":{"1min":8.0359999999999996,"5min":6.5289999999999999,"15min":6.5289999999999999},"max":{"1min":20.704000000000001,"5min":37.215000000000003,"15min":37.215000000000003},"last":14.958},{"interface":"front","average":{"1min":16.030999999999999,"5min":16.199000000000002,"15min":17.021999999999998},"min":{"1min":9.8300000000000001,"5min":8.7870000000000008,"15min":8.7870000000000008},"max":{"1min":24.986999999999998,"5min":37.753999999999998,"15min":37.753999999999998},"last":13.721}]},{"osd":7,"last update":"Fri Aug 30 16:05:24 2024","interfaces":[{"interface":"back","average":{"1min":13.443,"5min":14.657,"15min":15.507999999999999},"min":{"1min":7.9950000000000001,"5min":7.7130000000000001,"15min":7.7130000000000001},"max":{"1min":21.992999999999999,"5min":33.314,"15min":34.551000000000002},"last":16.66},{"interface":"front","average":{"1min":12.443,"5min":15.465,"15min":16.873999999999999},"min":{"1min":6.375,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":18.116,"5min":35.701999999999998,"15min":35.701999999999998},"last":12.779}]}]},{"osd":2,"up_from":16,"seq":68719476825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:17 2024","interfaces":[{"interface":"back","average":{"1min":10.505000000000001,"5min":9.4960000000000004,"15min":9.6129999999999995},"min":{"1min":4.1200000000000001,"5min":4.1200000000000001,"15min":3.6709999999999998},"max":{"1min":25.526,"5min":29.303000000000001,"15min":30.366},"last":8.5069999999999997},{"interface":"front","average":{"1min":11.006,"5min":9.6720000000000006,"15min":10.569000000000001},"min":{"1min":5.4009999999999998,"5min":4.5839999999999996,"15min":3.4900000000000002},"max":{"1min":25.300000000000001,"5min":27.916,"15min":45.604999999999997},"last":5.7539999999999996}]},{"osd":1,"last update":"Fri Aug 30 16:05:33 2024","interfaces":[{"interface":"back","average":{"1min":12.826000000000001,"5min":13.023999999999999,"15min":13.119999999999999},"min":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":41.786999999999999,"5min":41.786999999999999,"15min":46.460000000000001},"last":7.1920000000000002},{"interface":"front","average":{"1min":15.141,"5min":11.379,"15min":12.558999999999999},"min":{"1min":6.3650000000000002,"5min":5.944,"15min":5.944},"max":{"1min":67.332999999999998,"5min":67.332999999999998,"15min":67.332999999999998},"last":9.2929999999999993}]},{"osd":3,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":13.191000000000001,"5min":13.573,"15min":16.579000000000001},"min":{"1min":5.7850000000000001,"5min":5.5179999999999998,"15min":5.5179999999999998},"max":{"1min":31.547999999999998,"5min":71.475999999999999,"15min":71.475999999999999},"last":12.090999999999999},{"interface":"front","average":{"1min":14.112,"5min":13.813000000000001,"15min":18.425000000000001},"min":{"1min":6.218,"5min":6.218,"15min":6.218},"max":{"1min":40.046999999999997,"5min":40.046999999999997,"15min":50.564},"last":7.9729999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":15.548999999999999,"5min":14.409000000000001,"15min":17.457000000000001},"min":{"1min":8.4000000000000004,"5min":7.0970000000000004,"15min":7.0970000000000004},"max":{"1min":28.899000000000001,"5min":33.892000000000003,"15min":48.673999999999999},"last":10.48},{"interface":"front","average":{"1min":13.239000000000001,"5min":14.494,"15min":15.609999999999999},"min":{"1min":7.1879999999999997,"5min":6.7640000000000002,"15min":6.7640000000000002},"max":{"1min":27.448,"5min":67.054000000000002,"15min":67.054000000000002},"last":15.090999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:05:17 2024","interfaces":[{"interface":"back","average":{"1min":13.17,"5min":15.824,"15min":16.684999999999999},"min":{"1min":7.976,"5min":7.0259999999999998,"15min":4.7199999999999998},"max":{"1min":22.773,"5min":69.411000000000001,"15min":69.411000000000001},"last":9.843},{"interface":"front","average":{"1min":14.057,"5min":16.693000000000001,"15min":18.975000000000001},"min":{"1min":9.5449999999999999,"5min":8.6910000000000007,"15min":5.7460000000000004},"max":{"1min":24.228999999999999,"5min":56.152000000000001,"15min":56.917999999999999},"last":11.52}]},{"osd":6,"last update":"Fri Aug 30 16:03:27 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":16.623000000000001,"15min":18.198},"min":{"1min":10.302,"5min":7.4690000000000003,"15min":7.4690000000000003},"max":{"1min":23.317,"5min":72.75,"15min":72.75},"last":11.007999999999999},{"interface":"front","average":{"1min":19.774000000000001,"5min":18.396999999999998,"15min":20.119},"min":{"1min":9.8879999999999999,"5min":8.1579999999999995,"15min":8.1579999999999995},"max":{"1min":70.462999999999994,"5min":74.119,"15min":74.119},"last":15.827999999999999}]},{"osd":7,"last update":"Fri Aug 30 16:05:28 2024","interfaces":[{"interface":"back","average":{"1min":13.157,"5min":17.632000000000001,"15min":20.654},"min":{"1min":8.3160000000000007,"5min":8.2639999999999993,"15min":8.2639999999999993},"max":{"1min":21.460000000000001,"5min":137.33000000000001,"15min":137.33000000000001},"last":16.550999999999998},{"interface":"front","average":{"1min":15.618,"5min":19.195,"15min":19.751000000000001},"min":{"1min":9.7880000000000003,"5min":7.2770000000000001,"15min":7.2770000000000001},"max":{"1min":30.722999999999999,"5min":136.66399999999999,"15min":136.66399999999999},"last":14.124000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509528,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":7.3120000000000003,"5min":10.007,"15min":9.7769999999999992},"min":{"1min":4.8090000000000002,"5min":4.7489999999999997,"15min":4.7489999999999997},"max":{"1min":14.276,"5min":40.735999999999997,"15min":40.735999999999997},"last":7.2389999999999999},{"interface":"front","average":{"1min":10.282,"5min":11.349,"15min":11.352},"min":{"1min":5.0069999999999997,"5min":4.383,"15min":4.383},"max":{"1min":25.077999999999999,"5min":42.387,"15min":42.387},"last":9.8469999999999995}]},{"osd":2,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":10.721,"5min":11.506,"15min":11.308},"min":{"1min":6.3899999999999997,"5min":5.3689999999999998,"15min":3.3370000000000002},"max":{"1min":24.170999999999999,"5min":46.768999999999998,"15min":46.768999999999998},"last":7.806},{"interface":"front","average":{"1min":13.676,"5min":13.199999999999999,"15min":14.022},"min":{"1min":6.8860000000000001,"5min":6.2249999999999996,"15min":5.702},"max":{"1min":31.361000000000001,"5min":43.921999999999997,"15min":43.921999999999997},"last":13.638999999999999}]},{"osd":3,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":11.460000000000001,"5min":13.585000000000001,"15min":21.405000000000001},"min":{"1min":7.702,"5min":6.2839999999999998,"15min":6.2839999999999998},"max":{"1min":29.998999999999999,"5min":38.219000000000001,"15min":200.155},"last":16.048999999999999},{"interface":"front","average":{"1min":15.455,"5min":16.989000000000001,"15min":26.460999999999999},"min":{"1min":10.507999999999999,"5min":7.0940000000000003,"15min":7.0940000000000003},"max":{"1min":28.146000000000001,"5min":44.229999999999997,"15min":267.03100000000001},"last":15.368}]},{"osd":4,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.647,"5min":13.015000000000001,"15min":15.007},"min":{"1min":6.891,"5min":6.0970000000000004,"15min":6.0970000000000004},"max":{"1min":14.891,"5min":31.611999999999998,"15min":56.200000000000003},"last":11.477},{"interface":"front","average":{"1min":13.050000000000001,"5min":15.571,"15min":15.683999999999999},"min":{"1min":9.3930000000000007,"5min":9.298,"15min":8.609},"max":{"1min":16.626999999999999,"5min":46.875999999999998,"15min":48.534999999999997},"last":13.045999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:05:14 2024","interfaces":[{"interface":"back","average":{"1min":14.43,"5min":15.682,"15min":16.082999999999998},"min":{"1min":9.8059999999999992,"5min":5.9199999999999999,"15min":5.6920000000000002},"max":{"1min":40.006,"5min":43.308,"15min":43.308},"last":12.087},{"interface":"front","average":{"1min":16.951000000000001,"5min":17.693000000000001,"15min":17.853999999999999},"min":{"1min":9.9100000000000001,"5min":9.2149999999999999,"15min":4.75},"max":{"1min":49.575000000000003,"5min":49.575000000000003,"15min":49.575000000000003},"last":16.734999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:03:32 2024","interfaces":[{"interface":"back","average":{"1min":13.808999999999999,"5min":14.292,"15min":14.385999999999999},"min":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"max":{"1min":39.460999999999999,"5min":59.328000000000003,"15min":59.328000000000003},"last":8.9909999999999997},{"interface":"front","average":{"1min":19.225000000000001,"5min":18.331,"15min":18.533999999999999},"min":{"1min":10.781000000000001,"5min":10.005000000000001,"15min":10.005000000000001},"max":{"1min":46.237000000000002,"5min":58.567,"15min":58.567},"last":14.253}]},{"osd":7,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":18.472000000000001,"5min":19.276,"15min":21.135999999999999},"min":{"1min":11.045,"5min":10.632999999999999,"15min":9.1180000000000003},"max":{"1min":57.804000000000002,"5min":57.804000000000002,"15min":59.515999999999998},"last":17.440000000000001},{"interface":"front","average":{"1min":11.272,"5min":13.384,"15min":14.976000000000001},"min":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":7.0590000000000002},"max":{"1min":14.231999999999999,"5min":37.113,"15min":58.807000000000002},"last":10.954000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574937,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":658493,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:20 2024","interfaces":[{"interface":"back","average":{"1min":6.2510000000000003,"5min":8.2520000000000007,"15min":12.516999999999999},"min":{"1min":4.0300000000000002,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":13.52,"5min":53.558999999999997,"15min":111.313},"last":5.7060000000000004},{"interface":"front","average":{"1min":8.1809999999999992,"5min":9.4659999999999993,"15min":13.936},"min":{"1min":4.266,"5min":4.266,"15min":4.266},"max":{"1min":14.064,"5min":52.691000000000003,"15min":107.937},"last":6.7009999999999996}]},{"osd":1,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":9.3030000000000008,"5min":10.489000000000001,"15min":12.541},"min":{"1min":5.4539999999999997,"5min":4.657,"15min":4.657},"max":{"1min":16.811,"5min":34.545999999999999,"15min":46.988},"last":9.1799999999999997},{"interface":"front","average":{"1min":10.688000000000001,"5min":12.9,"15min":15.188000000000001},"min":{"1min":7.5179999999999998,"5min":6.4710000000000001,"15min":6.4710000000000001},"max":{"1min":15.455,"5min":33.203000000000003,"15min":47.661000000000001},"last":7.9470000000000001}]},{"osd":2,"last update":"Fri Aug 30 16:05:20 2024","interfaces":[{"interface":"back","average":{"1min":13.909000000000001,"5min":15.346,"15min":17.995000000000001},"min":{"1min":8.5630000000000006,"5min":6.9569999999999999,"15min":4.7220000000000004},"max":{"1min":31.667000000000002,"5min":41.549999999999997,"15min":84.317999999999998},"last":12.19},{"interface":"front","average":{"1min":11.781000000000001,"5min":13.901,"15min":16.207000000000001},"min":{"1min":6.9829999999999997,"5min":6.2770000000000001,"15min":6.0279999999999996},"max":{"1min":29.847999999999999,"5min":43.314999999999998,"15min":99.884},"last":15.792}]},{"osd":3,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":12.699,"5min":16.114000000000001,"15min":16.768999999999998},"min":{"1min":9.6899999999999995,"5min":6.2149999999999999,"15min":6.2149999999999999},"max":{"1min":18.687000000000001,"5min":117.346,"15min":117.346},"last":8.6150000000000002},{"interface":"front","average":{"1min":12.143000000000001,"5min":14.827999999999999,"15min":15.944000000000001},"min":{"1min":6.5270000000000001,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":19.370999999999999,"5min":119.23099999999999,"15min":119.23099999999999},"last":11.151}]},{"osd":4,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":12.651999999999999,"5min":15.768000000000001,"15min":17.32},"min":{"1min":6.5599999999999996,"5min":6.5599999999999996,"15min":6.5599999999999996},"max":{"1min":20.619,"5min":33.582000000000001,"15min":38.095999999999997},"last":13.640000000000001},{"interface":"front","average":{"1min":12.183999999999999,"5min":14.702,"15min":16.603999999999999},"min":{"1min":6.71,"5min":6.71,"15min":6.71},"max":{"1min":20.678999999999998,"5min":31.367000000000001,"15min":40.649000000000001},"last":15.081}]},{"osd":6,"last update":"Fri Aug 30 16:03:26 2024","interfaces":[{"interface":"back","average":{"1min":9.516,"5min":10.686,"15min":11.025},"min":{"1min":6.7809999999999997,"5min":6.367,"15min":6.367},"max":{"1min":15.27,"5min":26.344999999999999,"15min":26.344999999999999},"last":10.558},{"interface":"front","average":{"1min":19.596,"5min":17.696999999999999,"15min":17.396000000000001},"min":{"1min":11.433,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":58.052999999999997,"5min":58.052999999999997,"15min":58.052999999999997},"last":14.284000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:05:32 2024","interfaces":[{"interface":"back","average":{"1min":13.638,"5min":15.441000000000001,"15min":17.472999999999999},"min":{"1min":7.8490000000000002,"5min":7.8490000000000002,"15min":7.8490000000000002},"max":{"1min":25.056999999999999,"5min":38.180999999999997,"15min":43.604999999999997},"last":16.547000000000001},{"interface":"front","average":{"1min":9.7629999999999999,"5min":11.205,"15min":16.343},"min":{"1min":5.3579999999999997,"5min":5.3579999999999997,"15min":5.3579999999999997},"max":{"1min":22.509,"5min":31.547999999999998,"15min":88.278000000000006},"last":12.728999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:06:16.996 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-08-30T15:59:22.564097+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=15, tm_min=59, tm_sec=22, tm_wday=4, tm_yday=243, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=16, tm_min=6, tm_sec=8, tm_wday=4, tm_yday=243, tm_isdst=0) 2024-08-30T16:06:16.996 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-08-30T15:59:22.564097+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=15, tm_min=59, tm_sec=22, tm_wday=4, tm_yday=243, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=16, tm_min=6, tm_sec=8, tm_wday=4, tm_yday=243, tm_isdst=0) 2024-08-30T16:06:16.996 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-08-30T15:59:22.564097+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=15, tm_min=59, tm_sec=22, tm_wday=4, tm_yday=243, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=16, tm_min=6, tm_sec=8, tm_wday=4, tm_yday=243, tm_isdst=0) 2024-08-30T16:06:16.996 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-08-30T15:59:22.564097+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=15, tm_min=59, tm_sec=22, tm_wday=4, tm_yday=243, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=16, tm_min=6, tm_sec=8, tm_wday=4, tm_yday=243, tm_isdst=0) 2024-08-30T16:06:16.997 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-08-30T15:59:22.564097+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=15, tm_min=59, tm_sec=22, tm_wday=4, tm_yday=243, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=16, tm_min=6, tm_sec=8, tm_wday=4, tm_yday=243, tm_isdst=0) 2024-08-30T16:06:16.997 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-08-30T15:59:22.564097+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=15, tm_min=59, tm_sec=22, tm_wday=4, tm_yday=243, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=16, tm_min=6, tm_sec=8, tm_wday=4, tm_yday=243, tm_isdst=0) 2024-08-30T16:06:16.997 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-08-30T15:59:22.564097+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=15, tm_min=59, tm_sec=22, tm_wday=4, tm_yday=243, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=16, tm_min=6, tm_sec=8, tm_wday=4, tm_yday=243, tm_isdst=0) 2024-08-30T16:06:16.997 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-08-30T15:59:22.564097+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=15, tm_min=59, tm_sec=22, tm_wday=4, tm_yday=243, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=16, tm_min=6, tm_sec=8, tm_wday=4, tm_yday=243, tm_isdst=0) 2024-08-30T16:06:16.997 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-08-30T15:59:09.163382+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=15, tm_min=59, tm_sec=9, tm_wday=4, tm_yday=243, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=30, tm_hour=16, tm_min=6, tm_sec=8, tm_wday=4, tm_yday=243, tm_isdst=0) 2024-08-30T16:06:16.997 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-08-30T16:06:33.728 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:56.333 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:33.731 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:56.337 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:33.748 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:56.353 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:33.785 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:54.945 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:33.786 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:55.322 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:33.787 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:54.946 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:33.787 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:55.324 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:33.802 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:02:55.338 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:33.803 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:54.962 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:36.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-30T16:06:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-08-30T16:06:37.585 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-08-30T16:06:37.597 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-08-30T16:06:35.944134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1591398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222120,"kb_used_data":5480,"kb_used_omap":15,"kb_used_meta":216560,"kb_avail":754752600,"statfs":{"total":773094113280,"available":772866662400,"internally_reserved":0,"allocated":5611520,"data_stored":2880942,"data_compressed":1558,"data_compressed_allocated":49152,"data_compressed_original":786432,"omap_allocated":15591,"internal_metadata":221758233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002801"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":65,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-08-30T16:06:20.193478+0000","last_change":"2024-08-30T16:06:20.193478+0000","last_active":"2024-08-30T16:06:20.193478+0000","last_peered":"2024-08-30T16:06:20.193478+0000","last_clean":"2024-08-30T16:06:20.193478+0000","last_became_active":"2024-08-30T16:04:11.370524+0000","last_became_peered":"2024-08-30T16:04:11.370524+0000","last_unstale":"2024-08-30T16:06:20.193478+0000","last_undegraded":"2024-08-30T16:06:20.193478+0000","last_fullsized":"2024-08-30T16:06:20.193478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T16:06:20.188655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T16:06:20.188655+0000","last_clean_scrub_stamp":"2024-08-30T16:06:20.188655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T18:07:03.279688+0000","scrub_duration":0.029016309000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-08-30T16:06:17.108561+0000","last_change":"2024-08-30T16:06:17.117926+0000","last_active":"2024-08-30T16:06:17.108561+0000","last_peered":"2024-08-30T16:06:17.108561+0000","last_clean":"2024-08-30T16:06:17.108561+0000","last_became_active":"2024-08-30T16:04:38.082821+0000","last_became_peered":"2024-08-30T16:04:38.082821+0000","last_unstale":"2024-08-30T16:06:17.108561+0000","last_undegraded":"2024-08-30T16:06:17.108561+0000","last_fullsized":"2024-08-30T16:06:17.108561+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T16:06:17.106423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T16:06:17.106423+0000","last_clean_scrub_stamp":"2024-08-30T16:06:17.106423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T03:56:30.942352+0000","scrub_duration":0.32710483099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":37,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-08-30T16:06:13.215272+0000","last_change":"2024-08-30T16:06:13.225242+0000","last_active":"2024-08-30T16:06:13.215272+0000","last_peered":"2024-08-30T16:06:13.215272+0000","last_clean":"2024-08-30T16:06:13.215272+0000","last_became_active":"2024-08-30T16:05:38.361240+0000","last_became_peered":"2024-08-30T16:05:38.361240+0000","last_unstale":"2024-08-30T16:06:13.215272+0000","last_undegraded":"2024-08-30T16:06:13.215272+0000","last_fullsized":"2024-08-30T16:06:13.215272+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T16:06:13.212926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T16:06:13.212926+0000","last_clean_scrub_stamp":"2024-08-30T16:06:13.212926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T21:16:33.695400+0000","scrub_duration":0.33246900099999999,"objects_trimmed":0,"snaptrim_duration":0.113228546,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-08-30T16:06:16.145273+0000","last_change":"2024-08-30T16:06:16.154232+0000","last_active":"2024-08-30T16:06:16.145273+0000","last_peered":"2024-08-30T16:06:16.145273+0000","last_clean":"2024-08-30T16:06:16.145273+0000","last_became_active":"2024-08-30T16:05:39.773479+0000","last_became_peered":"2024-08-30T16:05:39.773479+0000","last_unstale":"2024-08-30T16:06:16.145273+0000","last_undegraded":"2024-08-30T16:06:16.145273+0000","last_fullsized":"2024-08-30T16:06:16.145273+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T16:06:16.143062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T16:06:16.143062+0000","last_clean_scrub_stamp":"2024-08-30T16:06:16.143062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:13:39.181597+0000","scrub_duration":0.292895356,"objects_trimmed":0,"snaptrim_duration":0.047212131999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"130'3","reported_seq":156,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-08-30T16:06:18.319323+0000","last_change":"2024-08-30T16:06:18.321629+0000","last_active":"2024-08-30T16:06:18.319323+0000","last_peered":"2024-08-30T16:06:18.319323+0000","last_clean":"2024-08-30T16:06:18.319323+0000","last_became_active":"2024-08-30T16:05:39.776453+0000","last_became_peered":"2024-08-30T16:05:39.776453+0000","last_unstale":"2024-08-30T16:06:18.319323+0000","last_undegraded":"2024-08-30T16:06:18.319323+0000","last_fullsized":"2024-08-30T16:06:18.319323+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'3","last_scrub_stamp":"2024-08-30T16:06:18.318451+0000","last_deep_scrub":"130'3","last_deep_scrub_stamp":"2024-08-30T16:06:18.318451+0000","last_clean_scrub_stamp":"2024-08-30T16:06:18.318451+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T22:52:01.803803+0000","scrub_duration":0.30030096099999998,"objects_trimmed":0,"snaptrim_duration":0.105854842,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":151,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-08-30T16:06:18.383796+0000","last_change":"2024-08-30T16:06:18.392966+0000","last_active":"2024-08-30T16:06:18.383796+0000","last_peered":"2024-08-30T16:06:18.383796+0000","last_clean":"2024-08-30T16:06:18.383796+0000","last_became_active":"2024-08-30T16:05:38.358584+0000","last_became_peered":"2024-08-30T16:05:38.358584+0000","last_unstale":"2024-08-30T16:06:18.383796+0000","last_undegraded":"2024-08-30T16:06:18.383796+0000","last_fullsized":"2024-08-30T16:06:18.383796+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T16:06:18.381716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T16:06:18.381716+0000","last_clean_scrub_stamp":"2024-08-30T16:06:18.381716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T20:13:58.691504+0000","scrub_duration":0.22087682,"objects_trimmed":0,"snaptrim_duration":0.069463435000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":153,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-08-30T16:06:15.333999+0000","last_change":"2024-08-30T16:06:15.343860+0000","last_active":"2024-08-30T16:06:15.333999+0000","last_peered":"2024-08-30T16:06:15.333999+0000","last_clean":"2024-08-30T16:06:15.333999+0000","last_became_active":"2024-08-30T16:05:38.358598+0000","last_became_peered":"2024-08-30T16:05:38.358598+0000","last_unstale":"2024-08-30T16:06:15.333999+0000","last_undegraded":"2024-08-30T16:06:15.333999+0000","last_fullsized":"2024-08-30T16:06:15.333999+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T16:06:15.331725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T16:06:15.331725+0000","last_clean_scrub_stamp":"2024-08-30T16:06:15.331725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-01T01:05:18.992436+0000","scrub_duration":0.25392089699999998,"objects_trimmed":0,"snaptrim_duration":0.082259062999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"130'2","reported_seq":74,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-08-30T16:06:19.543229+0000","last_change":"2024-08-30T16:06:19.547860+0000","last_active":"2024-08-30T16:06:19.543229+0000","last_peered":"2024-08-30T16:06:19.543229+0000","last_clean":"2024-08-30T16:06:19.543229+0000","last_became_active":"2024-08-30T16:04:07.760364+0000","last_became_peered":"2024-08-30T16:04:07.760364+0000","last_unstale":"2024-08-30T16:06:19.543229+0000","last_undegraded":"2024-08-30T16:06:19.543229+0000","last_fullsized":"2024-08-30T16:06:19.543229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'2","last_scrub_stamp":"2024-08-30T16:06:19.542432+0000","last_deep_scrub":"130'2","last_deep_scrub_stamp":"2024-08-30T16:06:19.542432+0000","last_clean_scrub_stamp":"2024-08-30T16:06:19.542432+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T16:57:06.160690+0000","scrub_duration":0.39652569500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"130'78","reported_seq":175,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-08-30T16:06:15.743472+0000","last_change":"2024-08-30T16:06:15.759905+0000","last_active":"2024-08-30T16:06:15.743472+0000","last_peered":"2024-08-30T16:06:15.743472+0000","last_clean":"2024-08-30T16:06:15.743472+0000","last_became_active":"2024-08-30T16:05:38.360837+0000","last_became_peered":"2024-08-30T16:05:38.360837+0000","last_unstale":"2024-08-30T16:06:15.743472+0000","last_undegraded":"2024-08-30T16:06:15.743472+0000","last_fullsized":"2024-08-30T16:06:15.743472+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'78","last_scrub_stamp":"2024-08-30T16:06:15.741107+0000","last_deep_scrub":"130'78","last_deep_scrub_stamp":"2024-08-30T16:06:15.741107+0000","last_clean_scrub_stamp":"2024-08-30T16:06:15.741107+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T23:15:29.692416+0000","scrub_duration":0.63280254199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1591360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1591360,"data_compressed":3116,"data_compressed_allocated":98304,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":1,"up_from":22,"seq":94489280609,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":724992,"data_stored":658493,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.896000000000001,"5min":11.542,"15min":13.461},"min":{"1min":5.5519999999999996,"5min":4.8049999999999997,"15min":4.8049999999999997},"max":{"1min":25.129999999999999,"5min":51.027000000000001,"15min":57.177999999999997},"last":7.0330000000000004},{"interface":"front","average":{"1min":9.5020000000000007,"5min":11.342000000000001,"15min":12.673999999999999},"min":{"1min":4.5759999999999996,"5min":4.5759999999999996,"15min":4.5759999999999996},"max":{"1min":16.521000000000001,"5min":54.682000000000002,"15min":58.024999999999999},"last":8.8119999999999994}]},{"osd":2,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.157,"5min":10.984999999999999,"15min":10.614000000000001},"min":{"1min":6.0880000000000001,"5min":5.5780000000000003,"15min":5.5780000000000003},"max":{"1min":15.81,"5min":73.399000000000001,"15min":73.399000000000001},"last":13.445},{"interface":"front","average":{"1min":13.478,"5min":14.470000000000001,"15min":14.875},"min":{"1min":6.5640000000000001,"5min":6.0049999999999999,"15min":6.0049999999999999},"max":{"1min":30.704999999999998,"5min":75.120000000000005,"15min":75.120000000000005},"last":12.247}]},{"osd":3,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":13.753,"5min":15.361000000000001,"15min":16.896000000000001},"min":{"1min":9.1199999999999992,"5min":7.0410000000000004,"15min":7.0410000000000004},"max":{"1min":29.506,"5min":66.869,"15min":66.869},"last":10.307},{"interface":"front","average":{"1min":10.523999999999999,"5min":13.089,"15min":14.315},"min":{"1min":5.8959999999999999,"5min":5.8959999999999999,"15min":5.8959999999999999},"max":{"1min":16.196000000000002,"5min":104.38200000000001,"15min":104.38200000000001},"last":16.361999999999998}]},{"osd":4,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":11.037000000000001,"5min":14.484999999999999,"15min":14.68},"min":{"1min":7.5590000000000002,"5min":6.7279999999999998,"15min":6.7279999999999998},"max":{"1min":18.202999999999999,"5min":66.340999999999994,"15min":66.340999999999994},"last":15.763999999999999},{"interface":"front","average":{"1min":9.5540000000000003,"5min":12.513999999999999,"15min":12.881},"min":{"1min":6.774,"5min":6.774,"15min":6.774},"max":{"1min":14.335000000000001,"5min":29.818999999999999,"15min":29.818999999999999},"last":12.885}]},{"osd":5,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":13.954000000000001,"5min":15.705,"15min":16.510999999999999},"min":{"1min":7.21,"5min":7.21,"15min":7.21},"max":{"1min":31.545000000000002,"5min":55.784999999999997,"15min":55.784999999999997},"last":8.1859999999999999},{"interface":"front","average":{"1min":11.759,"5min":17.265999999999998,"15min":18.946000000000002},"min":{"1min":6.0999999999999996,"5min":6.0999999999999996,"15min":6.0999999999999996},"max":{"1min":16.829999999999998,"5min":68.004999999999995,"15min":68.004999999999995},"last":9.4540000000000006}]},{"osd":6,"last update":"Fri Aug 30 16:06:26 2024","interfaces":[{"interface":"back","average":{"1min":34.755000000000003,"5min":34.755000000000003,"15min":34.755000000000003},"min":{"1min":12.744,"5min":12.744,"15min":12.744},"max":{"1min":208.61199999999999,"5min":208.61199999999999,"15min":208.61199999999999},"last":11.605},{"interface":"front","average":{"1min":26.295000000000002,"5min":26.295000000000002,"15min":26.295000000000002},"min":{"1min":7.8120000000000003,"5min":7.8120000000000003,"15min":7.8120000000000003},"max":{"1min":257.23200000000003,"5min":257.23200000000003,"15min":257.23200000000003},"last":17.398}]},{"osd":7,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":16.931000000000001,"5min":17.739999999999998,"15min":19.425999999999998},"min":{"1min":10.352,"5min":8.2050000000000001,"15min":8.2050000000000001},"max":{"1min":34.345999999999997,"5min":68.950000000000003,"15min":68.950000000000003},"last":11.039},{"interface":"front","average":{"1min":13.039999999999999,"5min":16.266999999999999,"15min":18.443999999999999},"min":{"1min":9.4529999999999994,"5min":8.3640000000000008,"15min":8.3640000000000008},"max":{"1min":17.625,"5min":36.186,"15min":59.079000000000001},"last":18.202000000000002}]}]},{"osd":7,"up_from":21,"seq":90194313312,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":10.666,"5min":9.0359999999999996,"15min":8.9800000000000004},"min":{"1min":6.1059999999999999,"5min":4.3789999999999996,"15min":4.3789999999999996},"max":{"1min":41.404000000000003,"5min":41.404000000000003,"15min":41.404000000000003},"last":10.316000000000001},{"interface":"front","average":{"1min":8.4649999999999999,"5min":8.8219999999999992,"15min":8.9979999999999993},"min":{"1min":5.4119999999999999,"5min":4.2590000000000003,"15min":4.2590000000000003},"max":{"1min":15,"5min":28.077000000000002,"15min":28.077000000000002},"last":6.4029999999999996}]},{"osd":1,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":13.212999999999999,"5min":12.092000000000001,"15min":11.654999999999999},"min":{"1min":6.5099999999999998,"5min":5.7009999999999996,"15min":5.7009999999999996},"max":{"1min":25.875,"5min":40.799999999999997,"15min":83.581000000000003},"last":12.239000000000001},{"interface":"front","average":{"1min":9.6549999999999994,"5min":10.743,"15min":12.013999999999999},"min":{"1min":5.3979999999999997,"5min":5.3979999999999997,"15min":5.3979999999999997},"max":{"1min":15.99,"5min":38.768999999999998,"15min":81.722999999999999},"last":7.4770000000000003}]},{"osd":2,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":15.76,"5min":14.09,"15min":13.834},"min":{"1min":6.7270000000000003,"5min":5.4850000000000003,"15min":5.4850000000000003},"max":{"1min":86.873000000000005,"5min":86.873000000000005,"15min":86.873000000000005},"last":9.6340000000000003},{"interface":"front","average":{"1min":19.004000000000001,"5min":14.856999999999999,"15min":16.341000000000001},"min":{"1min":7.7320000000000002,"5min":6.8029999999999999,"15min":6.8029999999999999},"max":{"1min":89.247,"5min":89.247,"15min":89.247},"last":14.006}]},{"osd":3,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":14.340999999999999,"5min":14.795,"15min":16.242000000000001},"min":{"1min":8.3089999999999993,"5min":7.9299999999999997,"15min":7.9299999999999997},"max":{"1min":27.538,"5min":54.076999999999998,"15min":65.739999999999995},"last":8.9179999999999993},{"interface":"front","average":{"1min":10.961,"5min":11.863,"15min":12.757999999999999},"min":{"1min":5.2430000000000003,"5min":5.2430000000000003,"15min":5.2430000000000003},"max":{"1min":17.991,"5min":55.978000000000002,"15min":64.980000000000004},"last":10.989000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.551,"5min":12.961,"15min":12.775},"min":{"1min":7.2069999999999999,"5min":6.4880000000000004,"15min":6.4880000000000004},"max":{"1min":17.509,"5min":41.631,"15min":41.631},"last":8.3320000000000007},{"interface":"front","average":{"1min":16.247,"5min":15.372,"15min":16.901},"min":{"1min":7.8810000000000002,"5min":7.8810000000000002,"15min":7.8810000000000002},"max":{"1min":43.097000000000001,"5min":43.097000000000001,"15min":43.097000000000001},"last":11.601000000000001}]},{"osd":5,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":13.321,"5min":12.627000000000001,"15min":11.811},"min":{"1min":7.2939999999999996,"5min":6.2380000000000004,"15min":6.2309999999999999},"max":{"1min":22.981000000000002,"5min":37.372999999999998,"15min":37.372999999999998},"last":13.420999999999999},{"interface":"front","average":{"1min":13.939,"5min":15.307,"15min":16.395},"min":{"1min":9.4920000000000009,"5min":8.2870000000000008,"15min":8.2870000000000008},"max":{"1min":29.893999999999998,"5min":38.484000000000002,"15min":38.618000000000002},"last":12.871}]},{"osd":6,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":14.888999999999999,"5min":14.888999999999999,"15min":14.888999999999999},"min":{"1min":9.1440000000000001,"5min":9.1440000000000001,"15min":9.1440000000000001},"max":{"1min":26.050000000000001,"5min":26.050000000000001,"15min":26.050000000000001},"last":17.361999999999998},{"interface":"front","average":{"1min":14.996,"5min":14.996,"15min":14.996},"min":{"1min":11.420999999999999,"5min":11.420999999999999,"15min":11.420999999999999},"max":{"1min":29.716000000000001,"5min":29.716000000000001,"15min":29.716000000000001},"last":14.750999999999999}]}]},{"osd":6,"up_from":105,"seq":450971566097,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28208,"kb_used_data":680,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343632,"statfs":{"total":96636764160,"available":96607879168,"internally_reserved":0,"allocated":696320,"data_stored":260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4434,"internal_metadata":28176046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:06:24 2024","interfaces":[{"interface":"back","average":{"1min":11.461,"5min":11.461,"15min":11.461},"min":{"1min":4.7110000000000003,"5min":4.7110000000000003,"15min":4.7110000000000003},"max":{"1min":45.341999999999999,"5min":45.341999999999999,"15min":45.341999999999999},"last":6.9299999999999997},{"interface":"front","average":{"1min":11.74,"5min":11.74,"15min":11.74},"min":{"1min":5.0970000000000004,"5min":5.0970000000000004,"15min":5.0970000000000004},"max":{"1min":44.203000000000003,"5min":44.203000000000003,"15min":44.203000000000003},"last":14.875999999999999}]},{"osd":1,"last update":"Fri Aug 30 16:06:24 2024","interfaces":[{"interface":"back","average":{"1min":12.936,"5min":12.936,"15min":12.936},"min":{"1min":5.9950000000000001,"5min":5.9950000000000001,"15min":5.9950000000000001},"max":{"1min":34.588999999999999,"5min":34.588999999999999,"15min":34.588999999999999},"last":11.592000000000001},{"interface":"front","average":{"1min":14.33,"5min":14.33,"15min":14.33},"min":{"1min":6.3300000000000001,"5min":6.3300000000000001,"15min":6.3300000000000001},"max":{"1min":49.906999999999996,"5min":49.906999999999996,"15min":49.906999999999996},"last":14.125}]},{"osd":2,"last update":"Fri Aug 30 16:06:24 2024","interfaces":[{"interface":"back","average":{"1min":19.699999999999999,"5min":19.699999999999999,"15min":19.699999999999999},"min":{"1min":7.75,"5min":7.75,"15min":7.75},"max":{"1min":88.594999999999999,"5min":88.594999999999999,"15min":88.594999999999999},"last":15.894},{"interface":"front","average":{"1min":17.277999999999999,"5min":17.277999999999999,"15min":17.277999999999999},"min":{"1min":6.8810000000000002,"5min":6.8810000000000002,"15min":6.8810000000000002},"max":{"1min":85.081000000000003,"5min":85.081000000000003,"15min":85.081000000000003},"last":7.9960000000000004}]},{"osd":3,"last update":"Fri Aug 30 16:06:24 2024","interfaces":[{"interface":"back","average":{"1min":16.343,"5min":16.343,"15min":16.343},"min":{"1min":7.6840000000000002,"5min":7.6840000000000002,"15min":7.6840000000000002},"max":{"1min":52.901000000000003,"5min":52.901000000000003,"15min":52.901000000000003},"last":10.244},{"interface":"front","average":{"1min":15.617000000000001,"5min":15.617000000000001,"15min":15.617000000000001},"min":{"1min":6.3280000000000003,"5min":6.3280000000000003,"15min":6.3280000000000003},"max":{"1min":67.143000000000001,"5min":67.143000000000001,"15min":67.143000000000001},"last":12.163}]},{"osd":4,"last update":"Fri Aug 30 16:06:24 2024","interfaces":[{"interface":"back","average":{"1min":16.577999999999999,"5min":16.577999999999999,"15min":16.577999999999999},"min":{"1min":7.835,"5min":7.835,"15min":7.835},"max":{"1min":56.740000000000002,"5min":56.740000000000002,"15min":56.740000000000002},"last":17.652999999999999},{"interface":"front","average":{"1min":16.390999999999998,"5min":16.390999999999998,"15min":16.390999999999998},"min":{"1min":9.4060000000000006,"5min":9.4060000000000006,"15min":9.4060000000000006},"max":{"1min":53.442999999999998,"5min":53.442999999999998,"15min":53.442999999999998},"last":12.893000000000001}]},{"osd":5,"last update":"Fri Aug 30 16:06:24 2024","interfaces":[{"interface":"back","average":{"1min":16.686,"5min":16.686,"15min":16.686},"min":{"1min":7.4249999999999998,"5min":7.4249999999999998,"15min":7.4249999999999998},"max":{"1min":67.707999999999998,"5min":67.707999999999998,"15min":67.707999999999998},"last":11.006},{"interface":"front","average":{"1min":18.558,"5min":18.558,"15min":18.558},"min":{"1min":9.0579999999999998,"5min":9.0579999999999998,"15min":9.0579999999999998},"max":{"1min":51.415999999999997,"5min":51.415999999999997,"15min":51.415999999999997},"last":18.41}]},{"osd":7,"last update":"Fri Aug 30 16:06:24 2024","interfaces":[{"interface":"back","average":{"1min":18.751999999999999,"5min":18.751999999999999,"15min":18.751999999999999},"min":{"1min":8.4930000000000003,"5min":8.4930000000000003,"15min":8.4930000000000003},"max":{"1min":68.233000000000004,"5min":68.233000000000004,"15min":68.233000000000004},"last":9.1859999999999999},{"interface":"front","average":{"1min":19.933,"5min":19.933,"15min":19.933},"min":{"1min":11.664,"5min":11.664,"15min":11.664},"max":{"1min":54.750999999999998,"5min":54.750999999999998,"15min":54.750999999999998},"last":16.521999999999998}]}]},{"osd":3,"up_from":21,"seq":90194313312,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":696320,"data_stored":260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":9.1189999999999998,"5min":9.6820000000000004,"15min":10.757999999999999},"min":{"1min":4.4459999999999997,"5min":4.3810000000000002,"15min":4.29},"max":{"1min":13.201000000000001,"5min":46.146999999999998,"15min":50.500999999999998},"last":9.3789999999999996},{"interface":"front","average":{"1min":11.837,"5min":10.550000000000001,"15min":13.154999999999999},"min":{"1min":5.234,"5min":5.1360000000000001,"15min":5.1360000000000001},"max":{"1min":31.273,"5min":36.720999999999997,"15min":51.965000000000003},"last":8.3719999999999999}]},{"osd":1,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.288,"5min":13.039999999999999,"15min":17.175000000000001},"min":{"1min":7.3159999999999998,"5min":5.8680000000000003,"15min":5.8680000000000003},"max":{"1min":21.776,"5min":44.862000000000002,"15min":137.56700000000001},"last":9.9849999999999994},{"interface":"front","average":{"1min":11.087,"5min":11.882,"15min":15.315},"min":{"1min":6.3470000000000004,"5min":5.5720000000000001,"15min":5.0060000000000002},"max":{"1min":29.413,"5min":47.530999999999999,"15min":136.321},"last":7.774}]},{"osd":2,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":13.003,"5min":12.824,"15min":15.257999999999999},"min":{"1min":6.4199999999999999,"5min":5.9109999999999996,"15min":5.9109999999999996},"max":{"1min":32.728000000000002,"5min":43.869999999999997,"15min":73.831000000000003},"last":12.635999999999999},{"interface":"front","average":{"1min":15.265000000000001,"5min":12.276,"15min":14.837999999999999},"min":{"1min":7.9059999999999997,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":28.201000000000001,"5min":28.201000000000001,"15min":71.831999999999994},"last":15.018000000000001}]},{"osd":4,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":12.999000000000001,"5min":13.83,"15min":24.760999999999999},"min":{"1min":7.3579999999999997,"5min":6.9199999999999999,"15min":6.9199999999999999},"max":{"1min":20.631,"5min":43.235999999999997,"15min":342.34500000000003},"last":10.574999999999999},{"interface":"front","average":{"1min":14.429,"5min":14.818,"15min":18.599},"min":{"1min":7.6529999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"max":{"1min":30.702999999999999,"5min":63.633000000000003,"15min":127.857},"last":15.618}]},{"osd":5,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":13.537000000000001,"5min":14.273999999999999,"15min":15.33},"min":{"1min":7.5179999999999998,"5min":6.5,"15min":6.5},"max":{"1min":35.07,"5min":58.715000000000003,"15min":58.715000000000003},"last":18.167000000000002},{"interface":"front","average":{"1min":14.728999999999999,"5min":16.553999999999998,"15min":18.331},"min":{"1min":9.8599999999999994,"5min":7.5869999999999997,"15min":7.2370000000000001},"max":{"1min":21.721,"5min":58.212000000000003,"15min":58.212000000000003},"last":11.132}]},{"osd":6,"last update":"Fri Aug 30 16:06:25 2024","interfaces":[{"interface":"back","average":{"1min":17.628,"5min":17.628,"15min":17.628},"min":{"1min":9.2550000000000008,"5min":9.2550000000000008,"15min":9.2550000000000008},"max":{"1min":35.652000000000001,"5min":35.652000000000001,"15min":35.652000000000001},"last":16.914000000000001},{"interface":"front","average":{"1min":17.218,"5min":17.218,"15min":17.218},"min":{"1min":9.6809999999999992,"5min":9.6809999999999992,"15min":9.6809999999999992},"max":{"1min":25.751000000000001,"5min":25.751000000000001,"15min":25.751000000000001},"last":13.263}]},{"osd":7,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.253,"5min":18.727,"15min":23.728000000000002},"min":{"1min":11.196999999999999,"5min":7.3049999999999997,"15min":7.3049999999999997},"max":{"1min":24.385000000000002,"5min":91.646000000000001,"15min":177.596},"last":13.834},{"interface":"front","average":{"1min":16.099,"5min":18.239000000000001,"15min":24.605},"min":{"1min":10.813000000000001,"5min":6.891,"15min":6.891},"max":{"1min":27.655000000000001,"5min":101.158,"15min":231.03800000000001},"last":16.206}]}]},{"osd":4,"up_from":20,"seq":85899346015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":8.5050000000000008,"5min":9.0839999999999996,"15min":10.901999999999999},"min":{"1min":4.3940000000000001,"5min":4.3940000000000001,"15min":4.3940000000000001},"max":{"1min":14.705,"5min":35.052,"15min":35.817999999999998},"last":8.3290000000000006},{"interface":"front","average":{"1min":9.9130000000000003,"5min":9.4039999999999999,"15min":11.090999999999999},"min":{"1min":6.548,"5min":4.6500000000000004,"15min":4.6500000000000004},"max":{"1min":25.111999999999998,"5min":35.091999999999999,"15min":35.091999999999999},"last":15.263999999999999}]},{"osd":1,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":14.266999999999999,"5min":13.131,"15min":12.057},"min":{"1min":6.165,"5min":5.6879999999999997,"15min":4.7000000000000002},"max":{"1min":28.699999999999999,"5min":34.347999999999999,"15min":34.347999999999999},"last":7.0039999999999996},{"interface":"front","average":{"1min":12.144,"5min":12.119999999999999,"15min":12.298},"min":{"1min":6.8890000000000002,"5min":5.4829999999999997,"15min":5.4829999999999997},"max":{"1min":21.145,"5min":35.481000000000002,"15min":35.481000000000002},"last":14.484999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":10.432,"5min":9.6359999999999992,"15min":9.9489999999999998},"min":{"1min":6.0910000000000002,"5min":5.4279999999999999,"15min":5.4279999999999999},"max":{"1min":21.757000000000001,"5min":30.666,"15min":30.666},"last":9.407},{"interface":"front","average":{"1min":14.686999999999999,"5min":13.294,"15min":14.146000000000001},"min":{"1min":7.0990000000000002,"5min":7.0629999999999997,"15min":7.0629999999999997},"max":{"1min":22.199000000000002,"5min":29.079999999999998,"15min":30.629000000000001},"last":17.597999999999999}]},{"osd":3,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.273999999999999,"5min":11.407999999999999,"15min":12.161},"min":{"1min":7.1260000000000003,"5min":5.8010000000000002,"15min":5.8010000000000002},"max":{"1min":21.704000000000001,"5min":28.315999999999999,"15min":29.715},"last":10.813000000000001},{"interface":"front","average":{"1min":14.173999999999999,"5min":14.201000000000001,"15min":15.092000000000001},"min":{"1min":6.3700000000000001,"5min":6.3700000000000001,"15min":6.3700000000000001},"max":{"1min":24.802,"5min":27.321999999999999,"15min":31.73},"last":16.824999999999999}]},{"osd":5,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":13.311,"5min":13.673999999999999,"15min":12.086},"min":{"1min":7.6719999999999997,"5min":6.8639999999999999,"15min":6.5010000000000003},"max":{"1min":28.126000000000001,"5min":42.143000000000001,"15min":42.143000000000001},"last":7.6879999999999997},{"interface":"front","average":{"1min":13.061,"5min":13.698,"15min":14.101000000000001},"min":{"1min":8.7620000000000005,"5min":6.5579999999999998,"15min":6.5579999999999998},"max":{"1min":21.623999999999999,"5min":43.917000000000002,"15min":43.917000000000002},"last":11.452999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.809000000000001,"5min":17.809000000000001,"15min":17.809000000000001},"min":{"1min":6.8600000000000003,"5min":6.8600000000000003,"15min":6.8600000000000003},"max":{"1min":82.522999999999996,"5min":82.522999999999996,"15min":82.522999999999996},"last":16.093},{"interface":"front","average":{"1min":19.478000000000002,"5min":19.478000000000002,"15min":19.478000000000002},"min":{"1min":10.448,"5min":10.448,"15min":10.448},"max":{"1min":83.543999999999997,"5min":83.543999999999997,"15min":83.543999999999997},"last":12.417999999999999}]},{"osd":7,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.690999999999999,"5min":14.922000000000001,"15min":15.625},"min":{"1min":9.4740000000000002,"5min":7.7130000000000001,"15min":7.7130000000000001},"max":{"1min":35.411999999999999,"5min":35.411999999999999,"15min":35.411999999999999},"last":13.052},{"interface":"front","average":{"1min":14.02,"5min":14.555,"15min":16.637},"min":{"1min":8.7560000000000002,"5min":5.7110000000000003,"15min":5.7110000000000003},"max":{"1min":21.116,"5min":35.701999999999998,"15min":35.701999999999998},"last":12.061}]}]},{"osd":2,"up_from":16,"seq":68719476829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:06:19 2024","interfaces":[{"interface":"back","average":{"1min":12.622999999999999,"5min":9.9299999999999997,"15min":9.8100000000000005},"min":{"1min":5.3659999999999997,"5min":4.1200000000000001,"15min":3.6709999999999998},"max":{"1min":71.631,"5min":71.631,"15min":71.631},"last":5.7910000000000004},{"interface":"front","average":{"1min":10.683999999999999,"5min":9.7750000000000004,"15min":10.547000000000001},"min":{"1min":5.7539999999999996,"5min":4.5839999999999996,"15min":3.4900000000000002},"max":{"1min":36.712000000000003,"5min":36.712000000000003,"15min":45.604999999999997},"last":8.3689999999999998}]},{"osd":1,"last update":"Fri Aug 30 16:05:33 2024","interfaces":[{"interface":"back","average":{"1min":12.826000000000001,"5min":13.023999999999999,"15min":13.119999999999999},"min":{"1min":5.2320000000000002,"5min":5.2320000000000002,"15min":5.2320000000000002},"max":{"1min":41.786999999999999,"5min":41.786999999999999,"15min":46.460000000000001},"last":12.419},{"interface":"front","average":{"1min":15.141,"5min":11.379,"15min":12.558999999999999},"min":{"1min":6.3650000000000002,"5min":5.944,"15min":5.944},"max":{"1min":67.332999999999998,"5min":67.332999999999998,"15min":67.332999999999998},"last":9.1940000000000008}]},{"osd":3,"last update":"Fri Aug 30 16:06:28 2024","interfaces":[{"interface":"back","average":{"1min":16.259,"5min":14.342000000000001,"15min":16.457999999999998},"min":{"1min":6.7069999999999999,"5min":5.5179999999999998,"15min":5.5179999999999998},"max":{"1min":67.918999999999997,"5min":71.475999999999999,"15min":71.475999999999999},"last":9.7799999999999994},{"interface":"front","average":{"1min":16.233000000000001,"5min":14.427,"15min":18.125},"min":{"1min":6.9379999999999997,"5min":6.218,"15min":6.218},"max":{"1min":70.555999999999997,"5min":70.555999999999997,"15min":70.555999999999997},"last":10.503}]},{"osd":4,"last update":"Fri Aug 30 16:06:28 2024","interfaces":[{"interface":"back","average":{"1min":22.315999999999999,"5min":16.016999999999999,"15min":17.68},"min":{"1min":7.1600000000000001,"5min":7.0970000000000004,"15min":7.0970000000000004},"max":{"1min":79.405000000000001,"5min":79.405000000000001,"15min":79.405000000000001},"last":12.988},{"interface":"front","average":{"1min":20.895,"5min":16.114000000000001,"15min":15.925000000000001},"min":{"1min":9.0020000000000007,"5min":6.7640000000000002,"15min":6.7640000000000002},"max":{"1min":155.584,"5min":155.584,"15min":155.584},"last":11.779}]},{"osd":5,"last update":"Fri Aug 30 16:06:19 2024","interfaces":[{"interface":"back","average":{"1min":18.844000000000001,"5min":16.635000000000002,"15min":16.800000000000001},"min":{"1min":8.3190000000000008,"5min":7.0259999999999998,"15min":4.7199999999999998},"max":{"1min":69.953999999999994,"5min":69.953999999999994,"15min":69.953999999999994},"last":16.234000000000002},{"interface":"front","average":{"1min":19.777999999999999,"5min":16.975000000000001,"15min":18.952000000000002},"min":{"1min":10.428000000000001,"5min":9.2100000000000009,"15min":5.7460000000000004},"max":{"1min":72.534000000000006,"5min":72.534000000000006,"15min":72.534000000000006},"last":14.129}]},{"osd":6,"last update":"Fri Aug 30 16:06:26 2024","interfaces":[{"interface":"back","average":{"1min":19.876999999999999,"5min":19.876999999999999,"15min":19.876999999999999},"min":{"1min":11.007999999999999,"5min":11.007999999999999,"15min":11.007999999999999},"max":{"1min":73.245999999999995,"5min":73.245999999999995,"15min":73.245999999999995},"last":13.590999999999999},{"interface":"front","average":{"1min":14.872999999999999,"5min":14.872999999999999,"15min":14.872999999999999},"min":{"1min":8.4969999999999999,"5min":8.4969999999999999,"15min":8.4969999999999999},"max":{"1min":26.353000000000002,"5min":26.353000000000002,"15min":26.353000000000002},"last":17.071000000000002}]},{"osd":7,"last update":"Fri Aug 30 16:06:28 2024","interfaces":[{"interface":"back","average":{"1min":17.614999999999998,"5min":16.241,"15min":20.350000000000001},"min":{"1min":9.5129999999999999,"5min":8.2639999999999993,"15min":8.2639999999999993},"max":{"1min":29.675000000000001,"5min":48.466000000000001,"15min":137.33000000000001},"last":17.831},{"interface":"front","average":{"1min":19.937999999999999,"5min":18.617000000000001,"15min":19.745000000000001},"min":{"1min":9.1839999999999993,"5min":7.2770000000000001,"15min":7.2770000000000001},"max":{"1min":69.132999999999996,"5min":69.132999999999996,"15min":136.66399999999999},"last":11.222}]}]},{"osd":0,"up_from":15,"seq":64424509532,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":692224,"data_stored":260653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":7.3120000000000003,"5min":10.007,"15min":9.7769999999999992},"min":{"1min":4.8090000000000002,"5min":4.7489999999999997,"15min":4.7489999999999997},"max":{"1min":14.276,"5min":40.735999999999997,"15min":40.735999999999997},"last":7.9199999999999999},{"interface":"front","average":{"1min":10.282,"5min":11.349,"15min":11.352},"min":{"1min":5.0069999999999997,"5min":4.383,"15min":4.383},"max":{"1min":25.077999999999999,"5min":42.387,"15min":42.387},"last":9.4600000000000009}]},{"osd":2,"last update":"Fri Aug 30 16:06:18 2024","interfaces":[{"interface":"back","average":{"1min":11.696,"5min":11.324999999999999,"15min":11.340999999999999},"min":{"1min":6.9020000000000001,"5min":5.3689999999999998,"15min":3.3370000000000002},"max":{"1min":22.286999999999999,"5min":30.539999999999999,"15min":46.768999999999998},"last":6.8070000000000004},{"interface":"front","average":{"1min":12.56,"5min":12.529999999999999,"15min":13.897},"min":{"1min":6.8940000000000001,"5min":6.2249999999999996,"15min":5.702},"max":{"1min":31.059999999999999,"5min":31.361000000000001,"15min":43.921999999999997},"last":13.582000000000001}]},{"osd":3,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":11.460000000000001,"5min":13.585000000000001,"15min":21.405000000000001},"min":{"1min":7.702,"5min":6.2839999999999998,"15min":6.2839999999999998},"max":{"1min":29.998999999999999,"5min":38.219000000000001,"15min":200.155},"last":11.848000000000001},{"interface":"front","average":{"1min":15.455,"5min":16.989000000000001,"15min":26.460999999999999},"min":{"1min":10.507999999999999,"5min":7.0940000000000003,"15min":7.0940000000000003},"max":{"1min":28.146000000000001,"5min":44.229999999999997,"15min":267.03100000000001},"last":14.891999999999999}]},{"osd":4,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":10.647,"5min":13.015000000000001,"15min":15.007},"min":{"1min":6.891,"5min":6.0970000000000004,"15min":6.0970000000000004},"max":{"1min":14.891,"5min":31.611999999999998,"15min":56.200000000000003},"last":8.827},{"interface":"front","average":{"1min":13.050000000000001,"5min":15.571,"15min":15.683999999999999},"min":{"1min":9.3930000000000007,"5min":9.298,"15min":8.609},"max":{"1min":16.626999999999999,"5min":46.875999999999998,"15min":48.534999999999997},"last":15.659000000000001}]},{"osd":5,"last update":"Fri Aug 30 16:06:18 2024","interfaces":[{"interface":"back","average":{"1min":13.827,"5min":15.134,"15min":15.919},"min":{"1min":7.9409999999999998,"5min":5.9199999999999999,"15min":5.6920000000000002},"max":{"1min":24.443999999999999,"5min":43.308,"15min":43.308},"last":12.381},{"interface":"front","average":{"1min":17.248000000000001,"5min":17.606000000000002,"15min":17.808},"min":{"1min":12.695,"5min":9.2880000000000003,"15min":4.75},"max":{"1min":26.021000000000001,"5min":49.575000000000003,"15min":49.575000000000003},"last":17.173999999999999}]},{"osd":6,"last update":"Fri Aug 30 16:06:27 2024","interfaces":[{"interface":"back","average":{"1min":13.558999999999999,"5min":13.558999999999999,"15min":13.558999999999999},"min":{"1min":6.7300000000000004,"5min":6.7300000000000004,"15min":6.7300000000000004},"max":{"1min":26.594000000000001,"5min":26.594000000000001,"15min":26.594000000000001},"last":10.17},{"interface":"front","average":{"1min":15.682,"5min":15.682,"15min":15.682},"min":{"1min":10.27,"5min":10.27,"15min":10.27},"max":{"1min":28.216000000000001,"5min":28.216000000000001,"15min":28.216000000000001},"last":12.914999999999999}]},{"osd":7,"last update":"Fri Aug 30 16:05:35 2024","interfaces":[{"interface":"back","average":{"1min":18.472000000000001,"5min":19.276,"15min":21.135999999999999},"min":{"1min":11.045,"5min":10.632999999999999,"15min":9.1180000000000003},"max":{"1min":57.804000000000002,"5min":57.804000000000002,"15min":59.515999999999998},"last":16.428999999999998},{"interface":"front","average":{"1min":11.272,"5min":13.384,"15min":14.976000000000001},"min":{"1min":7.0590000000000002,"5min":7.0590000000000002,"15min":7.0590000000000002},"max":{"1min":14.231999999999999,"5min":37.113,"15min":58.807000000000002},"last":10.696999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574941,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":724992,"data_stored":658493,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 30 16:06:23 2024","interfaces":[{"interface":"back","average":{"1min":7.859,"5min":8.1329999999999991,"15min":12.064},"min":{"1min":4.4560000000000004,"5min":3.9159999999999999,"15min":3.9159999999999999},"max":{"1min":13.292,"5min":53.558999999999997,"15min":111.313},"last":5.4790000000000001},{"interface":"front","average":{"1min":9.2010000000000005,"5min":9.4649999999999999,"15min":13.471},"min":{"1min":4.0279999999999996,"5min":4.0279999999999996,"15min":4.0279999999999996},"max":{"1min":18.736999999999998,"5min":52.691000000000003,"15min":107.937},"last":12.862}]},{"osd":1,"last update":"Fri Aug 30 16:06:34 2024","interfaces":[{"interface":"back","average":{"1min":32.652000000000001,"5min":14.938000000000001,"15min":13.813000000000001},"min":{"1min":6.0279999999999996,"5min":4.657,"15min":4.657},"max":{"1min":210.41,"5min":210.41,"15min":210.41},"last":7.8760000000000003},{"interface":"front","average":{"1min":33.902999999999999,"5min":16.859000000000002,"15min":16.359000000000002},"min":{"1min":7.9470000000000001,"5min":6.4710000000000001,"15min":6.4710000000000001},"max":{"1min":186.81100000000001,"5min":186.81100000000001,"15min":186.81100000000001},"last":12.180999999999999}]},{"osd":2,"last update":"Fri Aug 30 16:06:23 2024","interfaces":[{"interface":"back","average":{"1min":15.337,"5min":15.138999999999999,"15min":17.728999999999999},"min":{"1min":7.6369999999999996,"5min":7.2720000000000002,"15min":4.7220000000000004},"max":{"1min":36.777999999999999,"5min":41.549999999999997,"15min":84.317999999999998},"last":14.276},{"interface":"front","average":{"1min":15.688000000000001,"5min":14.183999999999999,"15min":16.094999999999999},"min":{"1min":6.7290000000000001,"5min":6.6689999999999996,"15min":6.0279999999999996},"max":{"1min":42.667999999999999,"5min":43.314999999999998,"15min":99.884},"last":8.7669999999999995}]},{"osd":3,"last update":"Fri Aug 30 16:06:34 2024","interfaces":[{"interface":"back","average":{"1min":25.190999999999999,"5min":16.544,"15min":17.309000000000001},"min":{"1min":8.1150000000000002,"5min":6.2149999999999999,"15min":6.2149999999999999},"max":{"1min":175.62100000000001,"5min":175.62100000000001,"15min":175.62100000000001},"last":15.632999999999999},{"interface":"front","average":{"1min":26.120000000000001,"5min":15.808,"15min":16.585000000000001},"min":{"1min":9.375,"5min":6.3600000000000003,"15min":6.3600000000000003},"max":{"1min":195.43899999999999,"5min":195.43899999999999,"15min":195.43899999999999},"last":9.375}]},{"osd":4,"last update":"Fri Aug 30 16:06:34 2024","interfaces":[{"interface":"back","average":{"1min":15.878,"5min":15.137,"15min":17.172999999999998},"min":{"1min":10.32,"5min":6.5599999999999996,"15min":6.5599999999999996},"max":{"1min":24.959,"5min":33.582000000000001,"15min":38.095999999999997},"last":13.619},{"interface":"front","average":{"1min":17.465,"5min":14.677,"15min":16.597999999999999},"min":{"1min":9.8900000000000006,"5min":6.71,"15min":6.71},"max":{"1min":29.010999999999999,"5min":31.367000000000001,"15min":40.649000000000001},"last":9.8900000000000006}]},{"osd":6,"last update":"Fri Aug 30 16:06:29 2024","interfaces":[{"interface":"back","average":{"1min":8.4209999999999994,"5min":8.4209999999999994,"15min":8.4209999999999994},"min":{"1min":6.5129999999999999,"5min":6.5129999999999999,"15min":6.5129999999999999},"max":{"1min":12.647,"5min":12.647,"15min":12.647},"last":6.9989999999999997},{"interface":"front","average":{"1min":25.763000000000002,"5min":25.763000000000002,"15min":25.763000000000002},"min":{"1min":9.2929999999999993,"5min":9.2929999999999993,"15min":9.2929999999999993},"max":{"1min":190.178,"5min":190.178,"15min":190.178},"last":14.944000000000001}]},{"osd":7,"last update":"Fri Aug 30 16:06:34 2024","interfaces":[{"interface":"back","average":{"1min":17.367000000000001,"5min":15.815,"15min":17.398},"min":{"1min":9.7449999999999992,"5min":7.8490000000000002,"15min":7.8490000000000002},"max":{"1min":25.812999999999999,"5min":38.180999999999997,"15min":43.604999999999997},"last":10.349},{"interface":"front","average":{"1min":10.502000000000001,"5min":11.021000000000001,"15min":15.782},"min":{"1min":6.9649999999999999,"5min":5.3579999999999997,"15min":5.3579999999999997},"max":{"1min":17.971,"5min":31.547999999999998,"15min":88.278000000000006},"last":10.91}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":397840,"data_compressed":779,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T16:06:37.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-08-30T16:06:38.405 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-30T16:06:38.405 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-30T16:06:38.406 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-08-30T16:06:38.406 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:06:38.416 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:06:38.407+0000 adad640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 22850) UID: 0 2024-08-30T16:06:38.416 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-08-30T16:06:38.408+0000 adad640 -1 osd.0 130 *** Got signal Terminated *** 2024-08-30T16:06:44.508 INFO:tasks.ceph.osd.0:Stopped 2024-08-30T16:06:44.508 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-08-30T16:06:44.509 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:06:44.517 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:06:44.508+0000 adad640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 22853) UID: 0 2024-08-30T16:06:44.517 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-08-30T16:06:44.508+0000 adad640 -1 osd.1 132 *** Got signal Terminated *** 2024-08-30T16:06:50.610 INFO:tasks.ceph.osd.1:Stopped 2024-08-30T16:06:50.611 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-08-30T16:06:50.611 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:06:50.619 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:06:50.611+0000 adad640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 22863) UID: 0 2024-08-30T16:06:50.622 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-08-30T16:06:50.611+0000 adad640 -1 osd.2 134 *** Got signal Terminated *** 2024-08-30T16:06:53.793 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:04:16.399 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:53.796 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:04:16.402 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:53.814 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:04:16.419 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:53.842 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:03:15.378 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:53.843 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:03:15.380 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:53.849 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:15.008 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:53.850 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:15.010 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:53.859 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:03:15.395 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:53.868 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:15.027 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:06:56.713 INFO:tasks.ceph.osd.2:Stopped 2024-08-30T16:06:56.713 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-08-30T16:06:56.714 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:06:56.723 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:06:56.713+0000 adad640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 22870) UID: 0 2024-08-30T16:06:56.725 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-08-30T16:06:56.714+0000 adad640 -1 osd.3 136 *** Got signal Terminated *** 2024-08-30T16:07:02.815 INFO:tasks.ceph.osd.3:Stopped 2024-08-30T16:07:02.816 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-08-30T16:07:02.816 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:07:02.825 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:07:02.815+0000 adad640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 23090) UID: 0 2024-08-30T16:07:02.825 INFO:tasks.ceph.osd.4.smithi094.stderr:2024-08-30T16:07:02.816+0000 adad640 -1 osd.4 139 *** Got signal Terminated *** 2024-08-30T16:07:08.918 INFO:tasks.ceph.osd.4:Stopped 2024-08-30T16:07:08.918 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-08-30T16:07:08.919 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:07:08.930 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:07:08.918+0000 adad640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 23093) UID: 0 2024-08-30T16:07:08.930 INFO:tasks.ceph.osd.5.smithi094.stderr:2024-08-30T16:07:08.918+0000 adad640 -1 osd.5 141 *** Got signal Terminated *** 2024-08-30T16:07:13.892 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:04:36.497 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:07:13.943 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:04:36.548 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:07:13.979 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:04:36.584 44239== Warning: unimplemented fcntl command: 1036 2024-08-30T16:07:13.997 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:35.156 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:07:13.998 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:35.157 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:07:14.006 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:03:35.543 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:07:14.010 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:03:35.547 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:07:14.017 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:35.176 58585== Warning: unimplemented fcntl command: 1036 2024-08-30T16:07:14.029 INFO:tasks.ceph.mon.b.smithi094.stderr:==00:00:03:35.565 40823== Warning: unimplemented fcntl command: 1036 2024-08-30T16:07:15.021 INFO:tasks.ceph.osd.5:Stopped 2024-08-30T16:07:15.022 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-08-30T16:07:15.022 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:07:15.034 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:07:15.024+0000 adad640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 45230) UID: 0 2024-08-30T16:07:15.034 INFO:tasks.ceph.osd.6.smithi094.stderr:2024-08-30T16:07:15.025+0000 adad640 -1 osd.6 145 *** Got signal Terminated *** 2024-08-30T16:07:15.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-08-30T16:07:21.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-08-30T16:07:21.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-08-30T16:07:21.124 INFO:tasks.ceph.osd.6:Stopped 2024-08-30T16:07:21.124 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-08-30T16:07:21.125 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:07:21.134 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:07:21.124+0000 adad640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 23098) UID: 0 2024-08-30T16:07:21.134 INFO:tasks.ceph.osd.7.smithi094.stderr:2024-08-30T16:07:21.124+0000 adad640 -1 osd.7 145 *** Got signal Terminated *** 2024-08-30T16:07:26.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-08-30T16:07:26.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-08-30T16:07:27.226 INFO:tasks.ceph.osd.7:Stopped 2024-08-30T16:07:27.227 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-30T16:07:27.227 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-08-30T16:07:27.227 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:07:27.252 INFO:tasks.ceph.mgr.y:Stopped 2024-08-30T16:07:27.252 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-08-30T16:07:27.252 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:07:27.253 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T16:07:27.252+0000 7f806d004640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 22553) UID: 0 2024-08-30T16:07:27.253 INFO:tasks.ceph.mgr.x.smithi094.stderr:2024-08-30T16:07:27.252+0000 7f806d004640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-08-30T16:07:27.281 INFO:tasks.ceph.mgr.x:Stopped 2024-08-30T16:07:27.282 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-30T16:07:27.282 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-30T16:07:27.282 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:07:27.308 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T16:07:27.295+0000 a084640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 44221) UID: 0 2024-08-30T16:07:27.311 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-08-30T16:07:27.297+0000 a084640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-08-30T16:07:31.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-08-30T16:07:31.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-08-30T16:07:33.383 INFO:tasks.ceph.mon.a:Stopped 2024-08-30T16:07:33.383 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-30T16:07:33.383 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:07:33.401 INFO:tasks.ceph.mon.c.smithi050.stderr:2024-08-30T16:07:33.387+0000 a084640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 58562) UID: 0 2024-08-30T16:07:33.402 INFO:tasks.ceph.mon.c.smithi050.stderr:2024-08-30T16:07:33.388+0000 a084640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-08-30T16:07:36.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-08-30T16:07:36.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-08-30T16:07:39.485 INFO:tasks.ceph.mon.c:Stopped 2024-08-30T16:07:39.486 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-30T16:07:39.486 INFO:teuthology.orchestra.run:waiting for 300 2024-08-30T16:07:39.502 INFO:tasks.ceph.mon.b.smithi094.stderr:2024-08-30T16:07:39.490+0000 a084640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 40821) UID: 0 2024-08-30T16:07:39.502 INFO:tasks.ceph.mon.b.smithi094.stderr:2024-08-30T16:07:39.491+0000 a084640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-08-30T16:07:41.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-08-30T16:07:41.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-08-30T16:07:45.588 INFO:tasks.ceph.mon.b:Stopped 2024-08-30T16:07:45.588 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-30T16:07:45.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(SLOW OPS\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-08-30T16:07:45.632 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi050.front.sepia.ceph.com 2024-08-30T16:07:45.633 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-08-30T16:07:45.772 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi050.front.sepia.ceph.com 2024-08-30T16:07:45.772 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-08-30T16:07:45.845 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi050.front.sepia.ceph.com 2024-08-30T16:07:45.845 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-08-30T16:07:45.903 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi050.front.sepia.ceph.com 2024-08-30T16:07:45.903 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-08-30T16:07:45.987 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi094.front.sepia.ceph.com 2024-08-30T16:07:45.987 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-08-30T16:07:46.106 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi094.front.sepia.ceph.com 2024-08-30T16:07:46.106 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-08-30T16:07:46.173 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi094.front.sepia.ceph.com 2024-08-30T16:07:46.173 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-08-30T16:07:46.230 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi094.front.sepia.ceph.com 2024-08-30T16:07:46.231 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-08-30T16:07:46.287 INFO:tasks.ceph:Archiving mon data... 2024-08-30T16:07:46.288 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi/7881803/data/mon.a.tgz 2024-08-30T16:07:46.288 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-08-30T16:07:46.306 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.mnFojUmuNE 2024-08-30T16:07:46.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.mnFojUmuNE 2024-08-30T16:07:46.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.mnFojUmuNE 2024-08-30T16:07:46.546 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.mnFojUmuNE is 432KB 2024-08-30T16:07:46.645 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.mnFojUmuNE 2024-08-30T16:07:46.660 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi/7881803/data/mon.c.tgz 2024-08-30T16:07:46.660 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-08-30T16:07:46.714 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.EAHEv8mN2f 2024-08-30T16:07:46.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.EAHEv8mN2f 2024-08-30T16:07:46.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.EAHEv8mN2f 2024-08-30T16:07:46.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-08-30T16:07:46.967 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.EAHEv8mN2f is 432KB 2024-08-30T16:07:47.170 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.EAHEv8mN2f 2024-08-30T16:07:47.186 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi/7881803/data/mon.b.tgz 2024-08-30T16:07:47.186 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-08-30T16:07:47.205 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.gpX5OyLxLd 2024-08-30T16:07:47.206 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.gpX5OyLxLd 2024-08-30T16:07:47.365 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.gpX5OyLxLd 2024-08-30T16:07:47.449 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.gpX5OyLxLd is 433KB 2024-08-30T16:07:47.545 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.gpX5OyLxLd 2024-08-30T16:07:47.561 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-30T16:07:47.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-30T16:07:47.563 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-30T16:07:47.626 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-08-30T16:07:47.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-08-30T16:07:47.631 DEBUG:teuthology.orchestra.run.smithi094:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-08-30T16:07:47.672 INFO:teuthology.orchestra.run.smithi050.stdout:/var/log/ceph/valgrind/mon.a.log: Leak_StillReachable 2024-08-30T16:07:47.672 INFO:teuthology.orchestra.run.smithi050.stdout:/var/log/ceph/valgrind/mon.c.log: Leak_StillReachable 2024-08-30T16:07:47.672 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind Leak_StillReachable 2024-08-30T16:07:47.672 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.a.log 2024-08-30T16:07:47.672 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind Leak_StillReachable 2024-08-30T16:07:47.672 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.c.log 2024-08-30T16:07:47.706 INFO:teuthology.orchestra.run.smithi094.stdout:/var/log/ceph/valgrind/mon.b.log: Leak_StillReachable 2024-08-30T16:07:47.706 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind Leak_StillReachable 2024-08-30T16:07:47.706 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.b.log 2024-08-30T16:07:47.706 INFO:tasks.ceph:Archiving crash dumps... 2024-08-30T16:07:47.708 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/crash to /home/teuthworker/archive/ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi/7881803/remote/smithi050/crash 2024-08-30T16:07:47.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-30T16:07:47.737 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/crash to /home/teuthworker/archive/ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi/7881803/remote/smithi094/crash 2024-08-30T16:07:47.738 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-30T16:07:47.770 INFO:tasks.ceph:Compressing logs... 2024-08-30T16:07:47.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-30T16:07:47.781 DEBUG:teuthology.orchestra.run.smithi094:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-30T16:07:49.117 INFO:tasks.ceph:Archiving logs... 2024-08-30T16:07:49.118 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi/7881803/remote/smithi050/log 2024-08-30T16:07:49.119 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-30T16:07:49.654 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi/7881803/remote/smithi094/log 2024-08-30T16:07:49.655 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-30T16:07:49.844 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e5753ca846ae583f991bb12d29862578d4f67590/qa/tasks/ceph.py", line 1924, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_e5753ca846ae583f991bb12d29862578d4f67590/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e5753ca846ae583f991bb12d29862578d4f67590/qa/tasks/ceph.py", line 362, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2024-08-30T16:07:49.846 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-30T16:07:49.859 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/__init__.py", line 641, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e5753ca846ae583f991bb12d29862578d4f67590/qa/tasks/ceph.py", line 1924, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_e5753ca846ae583f991bb12d29862578d4f67590/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e5753ca846ae583f991bb12d29862578d4f67590/qa/tasks/ceph.py", line 362, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2024-08-30T16:07:49.860 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-30T16:07:49.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-30T16:07:49.862 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-30T16:07:49.955 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-30T16:07:49.964 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-30T16:07:49.965 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-30T16:07:49.967 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-30T16:07:49.981 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-08-30T16:07:49.983 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-08-30T16:07:49.986 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-30T16:07:49.986 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-08-30T16:07:49.986 INFO:teuthology.orchestra.run.smithi050.stdout:^* hv01.front.sepia.ceph.com 3 6 377 3 +266us[ +262us] +/- 38ms 2024-08-30T16:07:49.986 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 -590us[ -594us] +/- 78ms 2024-08-30T16:07:49.986 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +65us[ +61us] +/- 75ms 2024-08-30T16:07:49.987 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T16:07:49.988 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-30T16:07:49.988 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-08-30T16:07:49.988 INFO:teuthology.orchestra.run.smithi094.stdout:^* hv01.front.sepia.ceph.com 3 6 377 4 +276us[ +275us] +/- 38ms 2024-08-30T16:07:49.989 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 7 -569us[ -571us] +/- 78ms 2024-08-30T16:07:49.989 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +76us[ +74us] +/- 75ms 2024-08-30T16:07:49.989 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T16:07:49.990 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-30T16:07:50.001 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-30T16:07:50.002 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-30T16:07:50.017 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-30T16:07:50.112 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-30T16:07:50.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-30T16:07:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T16:07:50.211 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-30T16:07:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T16:07:50.239 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-30T16:07:50.251 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-30T16:07:50.290 INFO:teuthology.task.internal:Duration was 1217.122627 seconds 2024-08-30T16:07:50.290 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-30T16:07:50.308 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-30T16:07:50.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-30T16:07:50.311 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-30T16:07:50.348 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-30T16:07:50.350 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-30T16:07:50.679 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-30T16:07:50.679 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2024-08-30T16:07:50.680 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-30T16:07:50.739 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2024-08-30T16:07:50.740 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-30T16:07:50.765 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-30T16:07:50.765 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-30T16:07:50.782 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-30T16:07:50.833 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-30T16:07:50.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-30T16:07:50.849 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-30T16:07:50.970 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-30T16:07:50.981 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-30T16:07:50.982 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-30T16:07:51.006 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-30T16:07:51.031 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-30T16:07:51.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-30T16:07:51.050 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-30T16:07:51.071 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2024-08-30T16:07:51.094 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2024-08-30T16:07:51.142 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-30T16:07:51.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T16:07:51.157 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-30T16:07:51.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T16:07:51.198 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-30T16:07:51.212 INFO:teuthology.task.internal:Transferring archived files... 2024-08-30T16:07:51.214 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi/7881803/remote/smithi050 2024-08-30T16:07:51.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-30T16:07:51.246 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/ksirivad-2024-08-30_14:53:30-rados-wip-ksirivad-backport-reef-57906-distro-default-smithi/7881803/remote/smithi094 2024-08-30T16:07:51.247 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-30T16:07:51.278 INFO:teuthology.task.internal:Removing archive directory... 2024-08-30T16:07:51.278 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-30T16:07:51.285 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-30T16:07:51.330 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-30T16:07:51.349 INFO:teuthology.task.internal:Not uploading archives. 2024-08-30T16:07:51.349 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-30T16:07:51.361 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-30T16:07:51.362 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-30T16:07:51.365 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-30T16:07:51.378 INFO:teuthology.orchestra.run.smithi050.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 30 16:07 /home/ubuntu/cephtest 2024-08-30T16:07:51.384 INFO:teuthology.orchestra.run.smithi094.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 30 16:07 /home/ubuntu/cephtest 2024-08-30T16:07:51.385 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-30T16:07:51.397 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-30T16:07:51.413 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: saw valgrind issues 2024-08-30T16:07:51.417 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/valgrind} duration: 1217.1226270198822 failure_reason: saw valgrind issues flavor: default owner: scheduled_ksirivad@teuthology status: fail success: false 2024-08-30T16:07:51.417 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-30T16:07:51.518 INFO:teuthology.run:FAIL